Dual-Anonymous Peer Review

Introduction

NASA's Science Mission Directorate (SMD) is strongly committed to promoting a culture that actively encourages diversity and inclusion and removes barriers to participation. One important way of achieving this objective is to ensure that the review of proposals is performed in an equitable and fair manner that reduces the impacts of any unconscious or implicit biases.

To this end, and motivated by a successful study conducted for the Hubble Space Telescope, SMD is conducting a pilot program in ROSES-2020 to evaluate proposals using dual-anonymous peer review (DAPR). Under this system, not only are proposers not told the identity their reviewers, the reviewers are not told the identity of the proposers, until after they have evaluated the scientific merit of all of the anonymized proposals.

ROSES-2020 Programs Taking Part in Dual-Anonymous Peer Review

Proposals submitted to the following ROSES-2020 elements will be evaluated using dual-anonymous peer review:

ROSES Element

Program

Notes

A.27

Earth Science US Participating Investigator

Proposal will be anonymized.

B.4

Heliophysics Guest Investigators

Step-1 and Step-2 proposals will be anonymized.

E.4

Habitable Worlds

Only Step-2 proposals will be anonymized.

D.2

Astrophysics Data Analysis

Proposal will be anonymized.

D.5

Swift Cycle 17

Only Phase-1 proposals will be anonymized.

D.6

Fermi Cycle 14

Only Phase-1 proposals will be anonymized.

D.9

NuSTAR Cycle 7

Only Phase-1 proposals will be anonymized.

D.10

TESS Cycle 4

Only Phase-1 proposals will be anonymized.

D.11

NICER Cycle 3

Only Phase-1 proposals will be anonymized.

General Guidance for Proposers

The text of each program element contains a section on the key anonymization requirements for that particular program. In addition, the NSPIRES page of each program element contains a document entitled “Guidelines for Anonymized Proposals,” that provides a comprehensive set of instructions for the preparation of anonymized proposals, as well as an overview of the review process.

Essential Points:

  1. Do not claim ownership of past work, e.g., "my previously funded work..." or "Our prior analysis demonstrates that…"
  2. Do not include the names of the personnel associated with the proposal or their organizational affiliations. This includes but is not limited to, page headers, footers, diagrams, figures, or watermarks. This does not include references to past work, which should be included whenever relevant (see below).
  3. References must be written in the form of a number in a square bracket, e.g. [1], which will then correspond to the full citation in the reference list.
  4. When citing references, use third person neutral wording. This especially applies to self-referencing. For example, replace phrases like "as we have shown in our previous work [17], …" with "as previously shown [17], ..."
  5. Depending on the program element, it may be occasionally important to cite exclusive access datasets, non-public software, unpublished data, or findings that have been presented in public before but are not citeable. Each of these may reveal (or strongly imply) the investigators on the proposal. In these instances, proposers must use language such "obtained in private communication" or "from private consultation" when referring to such potentially identifying work.

 

Preparation of the "Expertise and Resources – Not Anonymized" Document

Proposers will also be required to upload a separate "Expertise and Resources - Not Anonymized" document, that is not anonymized. Depending on the exact requirements of the program element, this document will contain most or all of the following elements:

  1. A list of all team members, together with their roles (e.g., PI, Co-I, collaborator).
  2. Brief descriptions of the scientific and technical expertise each team member brings, emphasizing the experiences necessary to be successful in executing the proposed work.
  3. The contribution that each team member will make to the proposed investigation.
  4. Specific resources (“Facilities and Equipment”, e.g., access to a laboratory, observatory, specific instrumentation, or specific samples or sites) that are required to perform the proposed investigation.
  5. A summary of work effort, to include the table of work effort. Given that the program element requires an anonymized version of this table in the main proposal body, the table here should be identical, but with the roles now also identified with names (e.g., Sandra Cauffman – PI; Nicky Fox – Co-I-1; Lori Glaze – Co-I-2).
  6. Bio sketches, if required by the program element.
  7. Statements of Current and Pending support, if required by the program element.
  8. Letters of resource support, if needed, see Table 1 of ROSES.


This "Expertise and Resources – Not Anonymized document will be distributed to the review panel only after all proposals have been reviewed and rated and only for proposals that are in the selectable range. This is to allow the reviewers to assess the team capabilities required to execute a given proposed science investigation.

 

Example Text for Anonymized Proposals

Much of the following text has been reproduced, with permission, from the Hubble Space Telescope dual-anonymous peer review website.

Here is an example of old fashioned not-Anonymized text from a sample proposal:

Over the last five years, we have used infrared photometry from 2MASS to compile a census of nearby ultracool M and L dwarfs (Cruz et al, 2003; 2006). We have identified 87 L dwarfs in 80 systems with nominal distances less than 20 parsecs from the Sun. This is the first true L dwarf census – a large-scale, volume-limited sample. Most distances are based on spectroscopic parallaxes, accurate to 20%, which is adequate for present purposes. Fifty systems already have high-resolution imaging, including our Cycle 9 and 13 snapshot programs, #8581 and #10143; nine are in binary or multiple systems, including six new discoveries. We propose to target the remaining sources via the current proposal.

Here is the same text, re-worked following the anonymizing guidelines:

Over the last five years, 2MASS infrared photometry has been used to compile a census of nearby ultracool M and L dwarfs [6,7]. 87 L dwarfs in 80 systems have been identified with nominal distances less than 20 parsecs from the Sun. This is the first true L dwarf census – a large-scale, volume-limited sample. Most distances are based on spectroscopic parallaxes, accurate to 20%, which is adequate for present purposes. Fifty systems already have high-resolution imaging, including the Cycle 9 and 13 snapshot programs, #8581 and #10143; nine are in binary or multiple systems, including six new discoveries. We propose to target the remaining sources via the current proposal.

Here is another example of text from a sample proposal:

In Rogers et al. (2014), we concluded that the best explanation for the dynamics of the shockwave and the spectra from both the forward-shocked ISM and the reverse-shocked ejecta is that a Type Ia supernova exploded into a preexisting wind-blown cavity. This object is the only known example of such a phenomenon, and it thus provides a unique opportunity to illuminate the nature of Type Ia supernovae and the progenitors. If our model from Rogers et al. (2014) is correct, then the single-degenerate channel for SNe Ia production must exist. We propose here for a second epoch of observations which we will compare with our first epoch obtained in 2007 to measure the proper motion of the shock wave.

Here is the same text, again re-worked following the anonymizing guidelines:

Prior work [12] concluded that the best explanation for the dynamics of the shockwave and the spectra from both the forward-shocked ISM and the reverse-shocked ejecta is that a Type Ia supernova exploded into a preexisting wind-blown cavity. This object is the only known example of such a phenomenon, and it thus provides a unique opportunity to illuminate the nature of Type Ia supernovae and the progenitors. If the model from [12] is correct, then the single-degenerate channel for SNe Ia production must exist. We propose here for a second epoch of observations which we will compare with a first epoch obtained in 2007 to measure the proper motion of the shock wave.

Another common situation that occurs in proposals is when a team member has institutional access to unique facilities (e.g., an observatory or laboratory) that are required to accomplish the proposed work. An anonymized proposal does not prohibit stating this fact in the Scientific/Technical/Management section of the proposal; however, the proposal must be written in a way that does not identify the team member. Here is an example:

The team has access to telescope time on the W. M. Keck Observatory, which will enable spectroscopic follow-up of the galaxies in the sample.

Note: in this situation, NASA strongly recommends that the team provide detailed supporting information to validate the claim in the “Expertise and Resources - Not Anonymized” document, that is not anonymized.

 

Guidance for Reviewers

The overarching objective of dual-anonymous peer review is to reduce unconscious bias in the evaluation of the merit of a proposal. In order to ensure this goal, the review panels will be instructed to evaluate proposal on their scientific merit without taking into account the proposing team qualifications. Here are some specific points.

  1. Consider proposals solely on the scientific merit of what is proposed.
  2. Do not spend any time attempting to identify the PI or the team. This applies even if you think you know the identities of the team members. Remember to discuss the science and not the people.
  3. In the panel discussions, do not make guesses on identities, insinuate the likely identities, or instigate discussion on a possible team’s past work.
  4. When writing evaluations, use neutral pronouns (e.g., "what they propose", or "the team has previously evaluated similar data").


As a final check, and only after the scientific evaluation is finalized for all proposals, the panel will be provided with the "Expertise and Resources - Not Anonymized" documents for those proposals in the selectable range, i.e., typically those rated "Good" or better. The panel will assess the qualifications of the team in order to allow the reviewers to assess the team capabilities required to execute a given proposed science investigation. If there are clear, compelling deficiencies in the expertise required to see through the goals of the proposal, the panel may note this in its comments to NASA. This review may not be used to “upgrade” proposals for having particularly strong team qualifications, nor may it be used to re-evaluate proposals. In addition, for those proposals that have an accompanying request for NASA's High End Computing resources, the HEC form will be released to reviewers at the same time.

 

Guidance for Levelers

NASA will appoint a "Leveler" to be present in the panel room for all discussions. The Leveler is not a reviewer or a panelist, but is an individual trained to ensure that the panel deliberations focus on the strengths and weaknesses of the proposal and do not deviate into a discussion of the identity, qualifications and experience of the PI and team.

Here are some specific points:

  1. Levelers are present to keep the panel discussions focused on scientific merit. Unlike the chairs, they are not listening for issues pertaining to the science, rather they are focused on the discussion itself.
  2. If the discussion veers to comments on the proposing team, their past work, their validity, or their identities, the Leveler’s job is to refocus that discussion.
  3. Levelers have the authority to stop the discussion on a proposal.

 

Dual-Anonymous Peer Review Questions and Answers

Last updated January 22, 2020

Q1. If I slip up in anonymizing my proposal, will it be returned without review?

A. NASA understands that dual-anonymous peer review represents a major shift in the evaluation of proposals, and as such there may be occasional slips in writing anonymized proposals. However, NASA reserves the right to return without review proposals that are particularly egregious in terms of the identification of the proposing team.

NASA further acknowledges that some proposed work may be so specialized that, despite attempts to anonymize the proposal, the identities of the Principal Investigator and team members are readily discernable. As long as the guidelines are followed, NASA will not return these proposals without review.

Please direct questions or corrections on this page to SARA@nasa.gov