All XRAS clients use a similar review interface. XRAS Review, hosted at URL similar to https://review-client.xras.org/, allows reviewers to view submissions and enter their reviews. The site also supports panel meetings with a sortable listing of submissions that can serve as an agenda or the more interactive Meeting App.

The Help item in the top navigation bar provides detailed instructions to help reviewers with the various XRAS Review components. In this section, we summarize the features and highlight aspects of particular interest to Administrators.

Accessing XRAS Review

XRAS Review is accessible only to persons you have made a member of one or more of your review panels. 

Reviewer Profile

To view and modify your XRAS reviewer profile, click your name in the upper right, then click User Profile. From this screen, you will be able to view all of your current and past panel memberships, as well as modify your Field(s) of Science expertise.

To add a primary or secondary field of science, click the blue “add” button, and select one or more fields of science from the list.

Impersonating Reviewers

A powerful aspect of XRAS is allowing an Admin to impersonate a reviewer in XRAS Review. By impersonating a reviewer, you can see that reviewer’s assignments, submit a review (e.g., if the reviewer is unable to access the review site), or report a conflict of interest on behalf of a reviewer.

This feature works much like the user impersonation feature in XRAS Submit and is only available if the “XRAS Review impersonator” permission has been specifically granted to an individual in XRAS Admin. At the right of the top navigation bar, click on your Admin username and enter the username of the reviewer you wish to impersonate. To stop impersonating, click on the reviewer username at top right and select “Stop Impersonating.”

Assignments

Upon logging into XRAS Review, reviewers are presented with their list of current review assignments and status, if any. Clicking on a row in the assignments table takes you to the review page for that submission.

If you are using XRAS’s multi-phase review capability, the Assignments table will show the review phase associated with each assignment.

Opportunities

The Opportunities menu allows the reviewer to select a current or recent Opportunity (within the last six months). Selecting an Opportunity takes you to a page with the list of submissions and meeting information, including associated documents, if any. Click on the provided PDF link to download a document. You can enter the meeting information and upload documents in the Edit Opportunity interface.

The list of submissions has been designed to serve as a basic agenda for a review panel meeting. The list can be sorted by clicking on the table headers and filtered by using the fields in the second row. Sort and filter options remain in effect if you click away from and later return to the table. Clicking anywhere in any other row will take you to the summary and review page for that submission. In the COIs column, hovering over a number will pop up a list of the reviewers who have a COI with that request.

XRAS uses color and text cues to indicate special situations for each reviewer:

• Purple shading and “(A)” in the Last Name column indicates the reviewer is assigned to a submission.

• Gray shading and “(C)” in the Last Name column indicate that the reviewer has a conflict of interest (COI) with the submission. XRAS does not allow a reviewer to click through to submissions for which they have a COI.

Rate Requests

The Rate Requests feature in XRAS allows reviewers to express their level of interest in rating the submissions for a given opportunity. You can control this optional feature on the Create/Edit Opportunity screen. 

Reviewers are presented with basic information about each request and allowed to select from a menu of options ranging from “Do not give it to me” to “Very interested in reviewing.” These preference ratings are visible to Admins via the Assign Reviewers interface.

If you are using multi-phase reviews, the Rate Requests listing will indicate for which phase a reviewer is rating their interests in reviewing allocations requests.

Your Review Documentation

You can configure XRAS Review to display a link to guidance for reviewers specific to your site’s allocations process, for example, your review criteria, rubrics, or policies. You can provide this link in the Client Settings.

Entering Reviews and Viewing Requests

The most detailed portion of XRAS Review is the set of information available for each request, including the form for entering reviews. Clicking on any submission in the list of requests for an Opportunity or from your list of review assignments will take you to the Review/View page for that request. All panel members without a conflict of interest can view this page; if a panel member is assigned to review the submission, they will also see the review entry form below the Summary. If the “Uploads Enabled” option is set on the Custom Review Forms tool, the user will be shown a tool that allows them to upload PDFs as part of their review. They will be able to view any review documents uploaded to this request and be able to delete documents they’ve uploaded.

The Request information page has a number of tabs that provide additional information, particularly if you have configured XRAS to integrate with your site’s local accounting service. The “XRAS Help” page describes the information displayed under each of the tabs.

Seeing Others’ Reviews

In XRAS, an assigned reviewer is able to see reviews by other panel members only after they have entered their own review. However, a reviewer may continue to edit their review as long as the Opportunity or submission remains “Under Review.” Panel members not assigned to review a submission can always see the reviews. 

Report COI

The Report COI tab lets a reviewer report a conflict of interest (COI) with a particular submission. While XRAS tries to automatically identify several common COIs, a reviewer may discover that they have a less common conflict type only after seeing the request details. Conflicts of interest are typically defined in the policies for your allocation process. You can add custom COI types under Client Settings.

From the Review/View page for a submission, a reviewer can select the “Report COI” tab, select the type of COI and provide a comment, then click the Submit COI button.

When a COI is reported, XRAS records the COI, clears the reviewer’s assignment (if applicable), and sends the reviewer back to their home Assignments screen. You can also configure XRAS to send an email notification to your Admin(s) that a COI was reported and review assignment was removed.

Meeting App

The Meeting App is a tool designed to help the flow of review meetings. It allows all users to see a list of requests and some quick details about selected requests. This includes the request title, the primary investigator, the request type, key personnel, conflicts of interest, the abstract, and the reviews for the request. It also allows admins to enter recommended amounts discussed at the meeting and non-admins to see the amounts entered by admins in real time.

The information page for an opportunity has a section at the top titled “Meeting Information.” If there is an upcoming or current meeting, a button titled "Review Meeting App" will be visible in this section. Clicking the button will open the Review Meeting App.

The app is divided into two main sections: the agenda on the left and the request details on the right. The request details section is divided into smaller sections. The first section contains the project title, which matches the format on the agenda, the request type, key personnel, conflicts of interest, reviewer instructions (if any), and the abstract. The reviewer instructions and abstract are hidden by default and can be expanded by clicking the “>” (right pointing arrow).

Agenda

The agenda contains all of the requests for the opportunity. Each entry will list the request number, the PI, a brief snippet of the abstract, any conflicts of interest, and any meeting review tags. Clicking on a request will load it for viewing in the Request Details section.

The agenda listing is also color coded, depending on the status of the request.

At the top of the agenda are filtering options, hidden by default until the magnifying glass icon is clicked. The list can be filtered by:

• The name of a reviewer

• The name of a conflict

• The name of the PI

• The name of a Resource

The filters can be combined to further narrow down the list.

Request Details

The request details section is divided into smaller sections. The first section contains the project title which matches the format on the agenda, the request type, key personnel, conflicts of interest, reviewer instructions (if any), and the abstract. The reviewer instructions and abstract are hidden by default and can be shown by clicking the “>” (right pointing arrow).

The next section contains the reviews. Each review is listed with the name of the reviewer, their overall evaluation, and a button to show the full review. The full review is hidden by default, clicking the “Full Review >” text will expand the review. The review will then show all of the fields for which the reviewer entered values.

The admin notes section is next and is only visible to and editable by your XRAS Admins. In this area, an admin can take notes about the request as well as see the notes that other admins have entered.

The tags section allows admins to click on relevant tags to assign to the request. Reviewers and other non-admins can only view which tags have been clicked.

The recommendations section lists the allocation begin and end dates, a list of the requested resources, the amount requested, and the amount recommended by the review panel. Non-admins can see the content of the allocation dates and the recommended amounts, but can't edit the contents. Admins can change the allocation dates, can enter numbers into the recommended amounts, and have the option to click the "Fill with Requested" shortcut to fill the recommended boxes with the requested amounts.

Lastly, a toggle and two save buttons are only available to admins. The toggle allows an admin to mark that the review is completed. The "Save" button will save all of the recommended amounts and tags to the database. The "Save Notes Only" will only save the admin notes; the recommended amounts will not be saved.

In the event that more than one admin has entered recommended amounts, allocation dates, or tags, the system will check for differences between the entries. If the another admin's entries are in conflict with any part of the current user's entry, a pop-up will appear before the current user's entries are saved. The user will see the items that are in conflict. The user can then choose to use the other admin's entries, use their own entries, or cancel the save altogether and return to the request view.