Dual-Anonymous Peer Review (DAPR)

Virtual Town Halls on Dual-Anonymous Peer Review

SMD-wide town hall: Dr. Thomas Zurbuchen, Associate Administrator, NASA’s Science Mission Directorate, hosted a virtual community town hall on March 3, 2020 to discuss the implementation of dual-anonymous peer review in ROSES-20. A recording of the town hall is below and the slides may be found here.

Astrophysics GO/GI: NASA's Astrophysics Division held a virtual community town hall on February 27, 2020 to discuss the implementation of dual-anonymous peer review for Astrophysics GI/GO programs (D.5, D.6, D.9-D.11). The slides from that presentation may be found here.

Astrophysics Data Analysis Program (ADAP): There was a special Webinar for Dual-Anonymous Peer Review for the Astrophysics Data Analysis program held on April 2, 2020. The slides from that webinar may be found here.

Earth Sciences USPI: On June 3, 2020, NASA’s Earth Sciences Division hosted a town hall to discuss the implementation of dual-anonymous peer review for the USPI program. The slides from that presentation may be found here.

Introduction

NASA's Science Mission Directorate (SMD) is strongly committed to promoting a culture that actively encourages diversity and inclusion and removes barriers to participation. One important way of achieving this objective is to ensure that the review of proposals is performed in an equitable and fair manner that reduces the impacts of any unconscious or implicit biases.

To this end, and motivated by a successful study conducted for the Hubble Space Telescope, SMD is conducting a pilot program in ROSES-2020 to evaluate proposals using dual-anonymous peer review (DAPR). Under this system, not only are proposers not told the identity their reviewers, the reviewers are not told the identity of the proposers, until after they have evaluated the scientific merit of all of the anonymized proposals.

ROSES-2020 Programs Taking Part in Dual-Anonymous Peer Review

Proposals submitted to the following ROSES-2020 elements will be evaluated using dual-anonymous peer review:

ROSES Element

Program

Notes

A.30

Earth Science US Participating Investigator

Proposal will be anonymized.

B.4

Heliophysics Guest Investigators

Step-1 and Step-2 proposals will be anonymized.

E.4

Habitable Worlds

Only Step-2 proposals will be anonymized.

D.2

Astrophysics Data Analysis

Proposal will be anonymized.

D.5

Swift Cycle 17

Only Phase-1 proposals will be anonymized.

D.6

Fermi Cycle 14

Only Phase-1 proposals will be anonymized.

D.9

NuSTAR Cycle 7

Only Phase-1 proposals will be anonymized.

D.10

TESS Cycle 4

Only Phase-1 proposals will be anonymized.

D.11

NICER Cycle 3

Only Phase-1 proposals will be anonymized.

General Guidance for Proposers

The text of each program element contains a section on the key anonymization requirements for that particular program. In addition, the NSPIRES page of each program element contains a document entitled “Guidelines for Anonymized Proposals,” that provides a comprehensive set of instructions for the preparation of anonymized proposals, as well as an overview of the review process.

Essential Points:

  1. Do not claim ownership of past work, e.g., "my previously funded work..." or "Our prior analysis demonstrates that…"
  2. Do not include the names of the personnel associated with the proposal or their organizational affiliations. This includes but is not limited to, page headers, footers, diagrams, figures, or watermarks. This does not include references to past work, which should be included whenever relevant (see below).
  3. References must be written in the form of a number in a square bracket, e.g. [1], which will then correspond to the full citation in the reference list.
  4. When citing references, use third person neutral wording. This especially applies to self-referencing. For example, replace phrases like "as we have shown in our previous work [17], …" with "as previously shown [17], ..."
  5. Depending on the program element, it may be occasionally important to cite exclusive access datasets, non-public software, unpublished data, or findings that have been presented in public before but are not citeable. Each of these may reveal (or strongly imply) the investigators on the proposal. In these instances, proposers must use language such "obtained in private communication" or "from private consultation" when referring to such potentially identifying work.

 

Preparation of the "Expertise and Resources – Not Anonymized" Document

Proposers will also be required to upload a separate "Expertise and Resources - Not Anonymized" document, that is not anonymized. Depending on the exact requirements of the program element, this document will contain most or all of the following elements:

  1. A list of all team members, together with their roles (e.g., PI, Co-I, collaborator).
  2. Brief descriptions of the scientific and technical expertise each team member brings, emphasizing the experiences necessary to be successful in executing the proposed work.
  3. The contribution that each team member will make to the proposed investigation.
  4. Specific resources (“Facilities and Equipment”, e.g., access to a laboratory, observatory, specific instrumentation, or specific samples or sites) that are required to perform the proposed investigation.
  5. A summary of work effort, to include the table of work effort. Given that the program element requires an anonymized version of this table in the main proposal body, the table here should be identical, but with the roles now also identified with names (e.g., Sandra Cauffman – PI; Nicky Fox – Co-I-1; Lori Glaze – Co-I-2).
  6. Bio sketches, if required by the program element.
  7. Statements of Current and Pending support, if required by the program element.
  8. Letters of resource support, if needed, see Table 1 of ROSES.


This “Expertise and Resources – Not Anonymized” document is distributed to the panel for a subset of proposals (typically the top third, according to the distribution of assigned grades and the projected selection rates). This is to allow the reviewers to assess the team capabilities required to execute a given proposed science investigation.

 

Example Text for Anonymized Proposals

Much of the following text has been reproduced, with permission, from the Hubble Space Telescope dual-anonymous peer review website.

Here is an example of old fashioned not-Anonymized text from a sample proposal:

Over the last five years, we have used infrared photometry from 2MASS to compile a census of nearby ultracool M and L dwarfs (Cruz et al, 2003; 2006). We have identified 87 L dwarfs in 80 systems with nominal distances less than 20 parsecs from the Sun. This is the first true L dwarf census – a large-scale, volume-limited sample. Most distances are based on spectroscopic parallaxes, accurate to 20%, which is adequate for present purposes. Fifty systems already have high-resolution imaging, including our Cycle 9 and 13 snapshot programs, #8581 and #10143; nine are in binary or multiple systems, including six new discoveries. We propose to target the remaining sources via the current proposal.

Here is the same text, re-worked following the anonymizing guidelines:

Over the last five years, 2MASS infrared photometry has been used to compile a census of nearby ultracool M and L dwarfs [6,7]. 87 L dwarfs in 80 systems have been identified with nominal distances less than 20 parsecs from the Sun. This is the first true L dwarf census – a large-scale, volume-limited sample. Most distances are based on spectroscopic parallaxes, accurate to 20%, which is adequate for present purposes. Fifty systems already have high-resolution imaging, including the Cycle 9 and 13 snapshot programs, #8581 and #10143; nine are in binary or multiple systems, including six new discoveries. We propose to target the remaining sources via the current proposal.

Here is another example of text from a sample proposal:

In Rogers et al. (2014), we concluded that the best explanation for the dynamics of the shockwave and the spectra from both the forward-shocked ISM and the reverse-shocked ejecta is that a Type Ia supernova exploded into a preexisting wind-blown cavity. This object is the only known example of such a phenomenon, and it thus provides a unique opportunity to illuminate the nature of Type Ia supernovae and the progenitors. If our model from Rogers et al. (2014) is correct, then the single-degenerate channel for SNe Ia production must exist. We propose here for a second epoch of observations which we will compare with our first epoch obtained in 2007 to measure the proper motion of the shock wave.

Here is the same text, again re-worked following the anonymizing guidelines:

Prior work [12] concluded that the best explanation for the dynamics of the shockwave and the spectra from both the forward-shocked ISM and the reverse-shocked ejecta is that a Type Ia supernova exploded into a preexisting wind-blown cavity. This object is the only known example of such a phenomenon, and it thus provides a unique opportunity to illuminate the nature of Type Ia supernovae and the progenitors. If the model from [12] is correct, then the single-degenerate channel for SNe Ia production must exist. We propose here for a second epoch of observations which we will compare with a first epoch obtained in 2007 to measure the proper motion of the shock wave.

Another common situation that occurs in proposals is when a team member has institutional access to unique facilities (e.g., an observatory or laboratory) that are required to accomplish the proposed work. An anonymized proposal does not prohibit stating this fact in the Scientific/Technical/Management section of the proposal; however, the proposal must be written in a way that does not identify the team member. Here is an example:

The team has access to telescope time on the W. M. Keck Observatory, which will enable spectroscopic follow-up of the galaxies in the sample.

Note: in this situation, NASA strongly recommends that the team provide detailed supporting information to validate the claim in the “Expertise and Resources - Not Anonymized” document, that is not anonymized.

 

Guidance for Reviewers

The overarching objective of dual-anonymous peer review is to reduce unconscious bias in the evaluation of the merit of a proposal. In order to ensure this goal, the review panels will be instructed to evaluate proposal on their scientific merit without taking into account the proposing team qualifications. Here are some specific points.

  1. Consider proposals solely on the scientific merit of what is proposed.
  2. Do not spend any time attempting to identify the PI or the team. This applies even if you think you know the identities of the team members. Remember to discuss the science and not the people.
  3. In the panel discussions, do not make guesses on identities, insinuate the likely identities, or instigate discussion on a possible team’s past work.
  4. When writing evaluations, use neutral pronouns (e.g., "what they propose", or "the team has previously evaluated similar data").


As a final check, and only after the scientific evaluation is finalized for all proposals, the panel will be provided with the "Expertise and Resources - Not Anonymized" documents for a subset of proposals (typically the top third, according to the distribution of assigned grades and the projected selection rates). The panel will assess the qualifications of the team in order to allow the reviewers to assess the team capabilities required to execute a given proposed science investigation. If there are clear, compelling deficiencies in the expertise required to see through the goals of the proposal, the panel may note this in its comments to NASA. This review may not be used to “upgrade” proposals for having particularly strong team qualifications, nor may it be used to re-evaluate proposals. In addition, for those proposals that have an accompanying request for NASA's High End Computing resources, the HEC form will be released to reviewers at the same time.

 

Guidance for Levelers

NASA will appoint a "Leveler" to be present in the panel room for all discussions. The Leveler is not a reviewer or a panelist, but is an individual trained to ensure that the panel deliberations focus on the strengths and weaknesses of the proposal and do not deviate into a discussion of the identity, qualifications and experience of the PI and team.

Here are some specific points:

  1. Levelers are present to keep the panel discussions focused on scientific merit. Unlike the chairs, they are not listening for issues pertaining to the science, rather they are focused on the discussion itself.
  2. If the discussion veers to comments on the proposing team, their past work, their validity, or their identities, the Leveler’s job is to refocus that discussion.
  3. Levelers have the authority to stop the discussion on a proposal.

 

Dual-Anonymous Peer Review Questions and Answers

Last updated July 8, 2020

Q1. If I slip up in anonymizing my proposal, will it be returned without review?

A1. NASA understands that dual-anonymous peer review represents a major shift in the evaluation of proposals, and as such there may be occasional slips in writing anonymized proposals. However, NASA reserves the right to return without review proposals that are particularly egregious in terms of the identification of the proposing team.

NASA further acknowledges that some proposed work may be so specialized that, despite attempts to anonymize the proposal, the identities of the Principal Investigator and team members are readily discernable. As long as the guidelines are followed, NASA will not return these proposals without review.

Q2: How will you deal with conflict of interest and bias?

A2: The NASA program scientists will check the names of the team members and participating organizations. During the discussion of the anonymized proposal, if the identities of the team members become evident to a reviewer, the reviewer must disclose to NASA if any strong biases exist that prevent the reviewer from delivering an objective assessment.

Q3: If the identity of the "proposing teams and institutions" is shrouded in secrecy, how then are proposing teams and institutions to discuss their track-record, ongoing work, complementary endeavors, institutional assets? For example, if an institution has been working closely with NASA for 40+ years on one specific topic (say, radar over ice), wouldn't all the programmatic, institutional, and PI experience that goes with that be lost from the review process?

A3: The anonymized proposal has no prohibition on discussing these aspects, merely that they be discussed without attribution to a particular investigator or group. In situations such as this, NASA recommends writing “previous work” instead of “our previous work”; or using “obtained in private communication”. Proposers should be able to make their case through their description of their proposed program of observations and analysis that they have the necessary skills to achieve success; if specific skills are required, the panel will flag that and will be able to verify this when they consult the “Expertise and Resources – Not Anonymized” document. The panel will provide a full analysis of the “Expertise and Resources – Not Anonymized” document and vote on using a three-point scale (uniquely qualified; qualified; not qualified).

Q4: While it is not possible for the proposing teams not to show any information in the proposals that might reveal their identities, such as the context and motivation of the proposed research, unique methodologies, and cited references, why keep the reviewers guessing who the proposers are (leading to undesirable consequences)? Furthermore, the track records of the proposers should be part of the merits of the proposals.

A4: It is entirely appropriate that the context and motivation of the research be addressed, as well as unique methodologies, references, etc. The main difference is that these aspects should be discussed without attribution to a particular investigator or group in the main body of the proposal. Remember that the goal of dual-anonymous peer review is to not make it completely impossible to guess the identities of the investigators, but to shift the focus of the discussion away from the individuals and toward the proposed science.

Q5: Assuming that the institution also has to be anonymous, how do reviewers determine if there are sufficient institutional resources to do the research?

A5: The track records of the proposing team will be addressed in the “Expertise and Resources – Not Anonymized” document and voted on using a three-point scale (uniquely qualified; qualified; not qualified).

Q6: What do you expect the unintended consequences of this action to be? Does this really serve the meritocracy?

A6: Experience with the Hubble Space Telescope dual-anonymous process indicates that there are few unintended consequences. However, NASA is proactively taking steps to ensure:

  • The SMD programs in the pilot lend themselves to dual-anonymous peer review.
  • Proposers have sufficient information and guidance to adequately anonymize their proposals.
  • Review panels are sufficiently briefed about dual-anonymous peer review.
  • The duration of each panel is not significantly increased.
  • High-risk/high-impact proposals are not disproportionally affected.

Q7: In the "Guidelines for Proposers to ROSES-2020 Dual-Anonymous Peer Review Programs" on page DAPR-6, it says that "generally one additional page is allotted for the Proposal Summary."  In "Heliophysics Guest Investigators - Open" (Appendix B.4) it says on page B.4-7 that the page limit is "10 pages for the central Science/Technical/Management section of proposal (that includes the one-page proposal summary above)." Do I understand correctly that the [text from the] anonymized Step-1 proposal is intended to be page 1 (and used as the proposal summary) of the 10-page technical proposal, which means that proposers have 9 pages (not 10) for the central section of their proposals?

A7: Yes, this is correct. However, please note that the “Expertise and Resources – Not Anonymized” document includes a single page for a description of team responsibilities, management structure, etc. (see Section 2.5.1. of the HGIO solicitation text). In other words, the total content length remains unchanged from previous HGIO competitions. Also note that the "Expertise and Resources – Not Anonymized" document also includes CVs, etc. as described in the "Guidelines for Anonymous Proposals" document. The “Expertise and Resources – Not Anonymized” has no overall page limit, but its constituent elements may themselves have page limits.

Q8: Is a table of contents permitted? I cannot find any mention of it in the DAPR guidance or in the text of B.4 HGIO.

A8: The Guidebook for proposers and the ROSES summary of solicitation set the default rules unless they are superseded by, e.g., B.1 The Heliophysics Research Program Overview or the program element. Since both the Guidebook for proposers and the ROSES summary of solicitation allow for a Table of Contents (of up to one page) this is permitted unless it is specifically prohibited. There is no mention of this in either B.1 or in B.4 Heliophysics Guest Investigators Open so it is permitted and it does not count against the page limited S/T/M section.

Q9: Even if the references are done by number [1, 2, etc.] and referencing is done in the 3rd person, there will be good number, maybe even dominant number, of references to my own prior work. So, it is very conceivable that a reviewer could guess by that who is proposing or at least from what group or institution the proposal comes.

A9: Yes, this is unavoidable to some extent. NASA chose square brackets in the text of the anonymized proposal so as not to keep reminding the reviewers who the proposers are likely to be during the reading of the proposal. However, absent a bibliography, the reviewers will have no means to assess any claims made in the proposal. Thus, we opted for a balanced way of handling references. One note: there may be a dominant number of references to a certain author’s work, but the dual-anonymous process still does not make it 100% clear who the PI is. For example, the PI could be a different or former member of the group cited frequently in the proposal.

Q10: Could you provide more information on the content of the budget and justification in the anonymous proposal?

A10: First and foremost, proposers should understand that there has been no change in the requirement that all information pertaining to salary levels, benefits, and overhead rates be redacted from the budget and budget justification included in the main proposal document.  This requirement is the same as it has been for the last several proposal cycles.  Beyond this requirement, under the dual-anonymous peer review process, proposers should also take care to anonymize the budget and budget narrative in the main proposal document by removing any language, logos, etc. that would reveal the names and/or institutions of the members of the proposing team.  Note that neither of these requirements--redaction and anonymization--apply to the "Total Budget" document, which is uploaded separately from the main proposal document and is not seen by the reviewers.

Please direct questions or corrections on this page to SARA@nasa.gov