Dual-Anonymous Peer Review (DAPR)
The appropriate DAPR Guidelines document will be posted under ‘Other Documents’ on the NSPIRES page for your ROSES program element.
Introduction
NASA’s Science Mission Directorate is strongly committed to ensuring that the review of proposals is performed in an equitable and fair manner. To this end, SMD will evaluate proposals to many ROSES program elements using dual-anonymous peer review (DAPR). Under this system, not only are proposers unaware of the identity of the members on the review panel, but the reviewers are not told the identities of the proposers until after the evaluation and rating of all proposals is complete (see below). The objective of dual-anonymous peer review is to minimize the impact of implicit or unconscious bias in the evaluation of the merit of a proposal.
The links to past Virtual Town Halls on Dual-Anonymous Peer Review that appear below may be useful for those who are already planning to submit a DAPR proposal and want to get into the details but for those who want to start by learning about the motivation and seeing the preliminary results may start with these Dual-Anonymous Peer Review (DAPR) Slides from a presentation by Dr. Hudgins.
The results of SMD’s pilot of dual-anonymous peer review in ROSES-2020 were consistent with improvements, both in terms of the overall quality of the review process, as well as in the demographics of awardees. For instance, in the ADAP program, prior to dual-anonymous review, women constituted 26% of the applicant pool, but only finished in the top two places in the panels’ rankings 16% of the time. Following the switch to dual-anonymous review, women constituted 31% of the pool and finished in the top two places 32% of the time. What’s more, the success rate of early-career investigators even eclipsed that of more seasoned investigators, further enriching the talent pool.
Given the positive response, the number of ROSES program elements have grown. While there is some uncertainty since many program elements are still TBD, we estimate that >40 program elements in ROSES-2024 will use DAPR.
Any program using DAPR will say so explicitly in the program element text. Moreover, the NSPIRES page of any program element using DAPR will host Guidelines on how to prepare proposals for Dual-Anonymous Peer Review under "Other documents". There are two separate instructions documents, one for most ROSES programs and another for proposals for phase-1 proposals to the Astrophysics General Investigator/Observer opportunities.
For Astrophysics General Investigator/Observer programs only the Phase-1 proposals must be anonymized. The default for ROSES is that (only) the full or Step-2 proposal must be anonymized. For Heliophysics (Appendix B), both the Step-1 and Step-2 must be anonymized. Read the instructions in the program element.
General Guidance for Proposers
The text of each program element contains a section on the key anonymization requirements for that particular program. In addition, the NSPIRES page of most DAPR program elements contain a document entitled “Guidelines for Anonymized Proposals,” that provides a comprehensive set of instructions for the preparation of anonymized proposals, as well as an overview of the review process.
Fill in all required information on the NSPIRES cover page (e.g., team members, institutions) not-anonymized. Except for the proposal summary, see below, the NSPIRES web “cover pages” (with Budget, Proposal Team and Program Specific and Business data, etc.) will be hidden from DAPR peer reviewers.
The Proposal Summary of a Step-2 or full proposal must be anonymized, omitting names of the team members or their institutions as well as any other individually-identifying information. The anonymized proposal summary shall be provided only as part of the NSPIRES cover page. Proposers should no longer include a copy of the anonymized proposal summary at the start of the uploaded proposal PDF file as was done in the past.
Essential Points:
Proposers are required to write the Scientific/Technical/Management (i.e., science justification) section of the proposal in an anonymized format, i.e., in a manner that does not explicitly identify the names of the team members or their institutions. Some specific points follow:
- Do not claim ownership of past work, e.g., "my previously funded work..." or "Our prior analysis demonstrates that…"
- Do not include the names of the personnel associated with the proposal or their organizational affiliations. This includes but is not limited to, page headers, footers, diagrams, figures, watermarks, or PDF bookmarks. This does not include references to past work, which should be included whenever relevant (see below).
- Do not associate personnel with named teams or collaborations, e.g., “the PI is a member of the EAGLE collaboration.”
- References must be written in the form of a number in a square bracket, e.g. [1], which will then correspond to the full citation in the reference list.
- When citing references, use third person neutral wording. This especially applies to self-referencing. For example, replace phrases like "as we have shown in our previous work [17], …" with "as previously shown [17], ..."
- Depending on the program element, it may be occasionally important to cite exclusive access datasets, non-public software, unpublished data, or findings that have been presented in public before but are not citable. Each of these may reveal (or strongly imply) the investigators on the proposal. In these instances, proposers must use language such "obtained in private communication" or "from private consultation" when referring to such potentially identifying work. If proposers include this type of citation, do not include with whom the personal communication took place, i.e., do not refer to the names or roles of individuals or provide a description of a team or group.
- As always, the reviewers expect proposers to describe the past work in the field to put the proposed work into context and how the proposed work would improve, build-upon, complement, contradict, or complete that past work. Using the above guidelines, proposers should be able to successfully accomplish this in an anonymized manner.
- The Summary of Work Effort and Budget are included in the proposal anonymized and again not anonymized in the separate "Expertise and Resources Not Anonymized" document.
- Omit any identifying institutional logos or insignia, e.g., that might appear on documents provided to you by your organization or subs, like budgets.
Most proposals to ROSES must include an "Open Science and Data Management Plan" (OSDMP), as part of the anonymized proposal document. In most cases, the OSDMP is included as a separate 2-page section of the anonymized proposal document, outside of the Scientific/Technical/Management (S/T/M) section. However, there are some program elements that require the OSDMP to be included within the page-limited S/T/M section so proposers should be careful to follow the instructions in the program element to which they are proposing. For more information on the OSDMP please see Section II.C of the ROSES Summary of Solicitation and https://science.nasa.gov/researchers/sara/faqs/OSDMP
Preparation of the "Expertise and Resources Not Anonymized" Document
Proposers will also be required to upload a separate "Expertise and Resources Not Anonymized" document, that is not anonymized. Depending on the exact requirements of the program element, this document will contain most or all of the following elements:
- A list of all team members, together with their roles (e.g., PI, Co-I, Collaborator) and institutional affiliations.
- Brief descriptions of the scientific and technical expertise each team member brings, emphasizing the areas of knowledge necessary to be successful in executing the proposed work.
- The contribution that each team member will make to the proposed investigation.
- Specific resources (“Facilities and Equipment”, e.g., access to a laboratory, observatory, specific instrumentation, or specific samples or sites) that are required to perform the proposed investigation.
- A summary of work effort, to include the table of work effort. Given that the program element requires an anonymized version of this table in the main proposal body, the table here should be identical, but with the roles now also identified with names (e.g., Sandra Cauffman – PI; Nicky Fox – Co-I-1; Lori Glaze – Co-I-2).
- Bio sketches, if required by the program element.
- Statements of Current and Pending support, if required by the program element.
- Letters of resource support, if needed, see Table 1 of ROSES.
This “Expertise and Resources Not Anonymized” document is distributed to the panel for a subset of proposals (typically the top third, according to the distribution of assigned grades and the projected selection rates). This is to allow the reviewers to assess the team capabilities required to execute a given proposed science investigation. This assessment will not affect the science grade and will be used by the Selection Official to help determine whether there would be any risks in funding the proposal.
Example Text for Anonymized Proposals
Much of the following text has been reproduced, with permission, from the Hubble Space Telescope dual-anonymous peer review website.
Here is an example of text from a sample proposal:
Over the last five years, we have used infrared photometry from 2MASS to compile a census of nearby ultracool M and L dwarfs (Cruz et al, 2003; 2006). We have identified 87 L dwarfs in 80 systems with nominal distances less than 20 parsecs from the Sun. This is the first true L dwarf census a large-scale, volume-limited sample. Most distances are based on spectroscopic parallaxes, accurate to 20%, which is adequate for present purposes. Fifty systems already have high-resolution imaging, including our Cycle 9 and 13 snapshot programs, #8581 and #10143; nine are in binary or multiple systems, including six new discoveries. We propose to target the remaining sources via the current proposal.
Here is the same text, re-worked following the anonymizing guidelines:
Over the last five years, 2MASS infrared photometry has been used to compile a census of nearby ultracool M and L dwarfs [6,7]. 87 L dwarfs in 80 systems have been identified with nominal distances less than 20 parsecs from the Sun. This is the first true L dwarf census a large-scale, volume-limited sample. Most distances are based on spectroscopic parallaxes, accurate to 20%, which is adequate for present purposes. Fifty systems already have high-resolution imaging available from two recent HST snapshot programs [REFERENCE]; nine are in binary or multiple systems, including six new discoveries. We propose to target the remaining sources via the current proposal.
Here is another example of text from a sample proposal:
In Rogers et al. (2014), we concluded that the best explanation for the dynamics of the shockwave and the spectra from both the forward-shocked ISM and the reverse-shocked ejecta is that a Type Ia supernova exploded into a preexisting wind-blown cavity. This object is the only known example of such a phenomenon, and it thus provides a unique opportunity to illuminate the nature of Type Ia supernovae and the progenitors. If our model from Rogers et al. (2014) is correct, then the single-degenerate channel for SNe Ia production must exist. We propose here for a second epoch of observations which we will compare with our first epoch obtained in 2007 to measure the proper motion of the shock wave.
Here is the same text, again re-worked following the anonymizing guidelines:
Prior work [12] concluded that the best explanation for the dynamics of the shockwave and the spectra from both the forward-shocked ISM and the reverse-shocked ejecta is that a Type Ia supernova exploded into a preexisting wind-blown cavity. This object is the only known example of such a phenomenon, and it thus provides a unique opportunity to illuminate the nature of Type Ia supernovae and the progenitors. If the model from [12] is correct, then the single-degenerate channel for SNe Ia production must exist. We propose here for a second epoch of observations which we will compare with a first epoch obtained in 2007 to measure the proper motion of the shock wave.
Here is a third example of text from a sample proposal:
Before and after radiolysis, we will test changes in ice composition with our established cryogenic mass spectrometry technique (2S-LAI-MS) [Henderson and Gudipati 2014; Henderson and Gudipati 2015]. Our technique uses an IR laser tuned to the absorption wavelength for water to gently eject the sample into the gas phase, where it can be ionized by a UV laser and analyzed by time-of-flight mass spectrometry. A key advantage of our technique is that compositional information can be obtained directly in situ, for temperatures that are relevant to Europa (i.e., 50, 100, 150 K), without a need for warming to room temperature or other sample preparation. We will also perform continuous mass spectral analyses (using a residual gas analyzer and a quadrupole mass spectrometer already installed) during radiation to quantify the amount of sputtered material and evolved gas byproducts.
Here is the same text, again re-worked following the anonymizing guidelines:
Before and after radiolysis, we will test changes in ice composition with an established cryogenic mass spectrometry technique [12,13]. This technique uses an IR laser tuned to the absorption wavelength for water to gently eject the sample into the gas phase, where it can be ionized by a UV laser and analyzed by time-of-flight mass spectrometry. A key advantage of this technique is that compositional information can be obtained directly in situ, for temperatures that are relevant to Europa (i.e., 50, 100, 150 K), without a need for warming to room temperature or other sample preparation. We will also perform continuous mass spectral analyses during radiation to quantify the amount of sputtered material and evolved gas byproducts.
Another common situation that occurs in proposals is when a team member has institutional access to unique facilities (e.g., access to a laboratory, observatory, specific instrumentation, or specific samples or sites) that are required to accomplish the proposed work. An anonymized proposal does not prohibit stating this fact in the Scientific/Technical/Management section of the proposal; however, the proposal must be written in a way that does not identify the team member. Here is an example:
"The team has been awarded XX days of telescope time on Keck to observe Titan" or "The team has XX days at the NASA Ames Vertical Gun Range to study impacts on Titan" or "The team has XX days in the NASA Venus In-situ Investigations Chamber, which will enable us to examine the properties of sulfuric acid rain on Venus."
Note: in this situation, NASA strongly recommends that the team provide detailed supporting information (e.g., a letter of resource support) to validate the claim in the "Expertise and Resources Not Anonymized" document.
Common Pitfalls in the Preparation of Anonymized Proposals
Below is a non-exhaustive list of common pitfalls when preparing anonymized proposals:
- Including metadata (e.g., PDF bookmarks, document properties) that reveal the name of the PI.
- Recycling proposals prepared prior to dual-anonymous peer review and not carefully anonymizing the text.
- Providing the names of investigators on the contents page or in a header or footer.
- Providing the origin of travel for professional travel (e.g., conferences).
- Mentioning the institution name in the Budget Narrative.
- Including the PI or Co-I names in budget tables.
- Attempting to “redact” identifying information by inserting a black rectangle over parts of the text, versus formally redacting the text using specialized software.
- Including the “Expertise and Resources Not Anonymized” document within the main proposal PDF.
Many of these issues may be resolved by carefully searching the proposal PDF for identifying information, e.g., PI name, Co-I name(s), institution(s) before submission.
Guidance for Reviewers
The overarching objective of dual-anonymous peer review is ensure that the review of proposals is performed in an equitable and fair manner. In order to ensure this goal, the review panels will be instructed to evaluate proposal on their scientific merit without taking into account the proposing team qualifications. Here are some specific points.
- Consider proposals solely on the scientific merit of what is proposed.
- Do not spend any time attempting to identify the PI or the team. This applies even if you think you know the identities of the team members. Remember to discuss the science and not the people.
- In the panel discussions, do not make guesses on identities, insinuate the likely identities, or instigate discussion on a possible team’s past work.
- When writing evaluations, use neutral pronouns (e.g., "what they propose", or "the team has previously evaluated similar data").
As a final check, and only after the scientific evaluation is finalized for all proposals, the panel will be provided with the "Expertise and Resources - Not Anonymized" documents for a subset of proposals (typically the top third, according to the distribution of assigned grades and the projected selection rates). The panel will assess the qualifications of the team in order to allow the reviewers to assess the team capabilities required to execute a given proposed science investigation. If there are clear, compelling deficiencies in the expertise required to see through the goals of the proposal, the panel may note this in its comments to NASA. This review may not be used to “upgrade” proposals for having particularly strong team qualifications, nor may it be used to re-evaluate proposals. In addition, for those proposals that have an accompanying request for NASA's High End Computing resources, the HEC form will be released to reviewers at the same time.
Guidance for Levelers
NASA will appoint a "Leveler" to be present in the panel room for all discussions. The Leveler is not a reviewer or a panelist, but is an individual trained to ensure that the panel deliberations focus on the strengths and weaknesses of the proposal and do not deviate into a discussion of the identity, qualifications and experience of the PI and team.
Here are some specific points:
- Levelers are present to keep the panel discussions focused on scientific merit. Unlike the chairs, they are not listening for issues pertaining to the science, rather they are focused on the discussion itself.
- If the discussion veers to comments on the proposing team, their past work, their validity, or their identities, the Leveler’s job is to refocus that discussion.
- Levelers have the authority to stop the discussion on a proposal.
Dual-Anonymous Peer Review Questions and Answers
Last updated March 2024
Q1. If I slip up in anonymizing my proposal, will it be returned without review?
A1. NASA understands that dual-anonymous peer review represents a major shift in the evaluation of proposals, and as such there may be occasional slips in writing anonymized proposals. However, NASA reserves the right to return without review proposals that are particularly egregious in terms of the identification of the proposing team.
NASA further acknowledges that some proposed work may be so specialized that, despite attempts to anonymize the proposal, the identities of the Principal Investigator and team members are readily discernable. As long as the guidelines are followed, NASA will not return these proposals without review.
Q2: How will you deal with conflict of interest and bias?
A2: The NASA program scientists will check the names of the team members and participating organizations. During the discussion of the anonymized proposal, if the identities of the team members become evident to a reviewer, the reviewer must disclose to NASA if any strong biases exist that prevent the reviewer from delivering an objective assessment.
Q3: If the identity of the "proposing teams and institutions" is shrouded in secrecy, how then are proposing teams and institutions to discuss their track-record, ongoing work, complementary endeavors, institutional assets? For example, if an institution has been working closely with NASA for 40+ years on one specific topic (say, radar over ice), wouldn't all the programmatic, institutional, and PI experience that goes with that be lost from the review process?
A3: The anonymized proposal has no prohibition on discussing these aspects, merely that they be discussed without attribution to a particular investigator or group. In situations such as this, NASA recommends writing “previous work” instead of “our previous work”; or using “obtained in private communication”. Proposers should be able to make their case through their description of their proposed program of observations and analysis that they have the necessary skills to achieve success; if specific skills are required, the panel will flag that and will be able to verify this when they consult the “Expertise and Resources Not Anonymized” document. The panel will provide a full analysis of the “Expertise and Resources Not Anonymized” document and vote on using a three-point scale.
Q4: While it is not possible for the proposing teams not to show any information in the proposals that might reveal their identities, such as the context and motivation of the proposed research, unique methodologies, and cited references, why keep the reviewers guessing who the proposers are (leading to undesirable consequences)? Furthermore, the track records of the proposers should be part of the merits of the proposals.
A4: It is entirely appropriate that the context and motivation of the research be addressed, as well as unique methodologies, references, etc. The main difference is that these aspects should be discussed without attribution to a particular investigator or group in the main body of the proposal. Remember that the goal of dual-anonymous peer review is to not make it completely impossible to guess the identities of the investigators, but to shift the focus of the discussion away from the individuals and toward the proposed science.
Q5: Assuming that the institution also has to be anonymous, how do reviewers determine if there are sufficient institutional resources to do the research?
A5: The track records of the proposing team will be addressed in the “Expertise and Resources – Not Anonymized” document and voted on using a three-point scale (uniquely qualified; qualified; not qualified).
Q6: What do you expect the unintended consequences of this action to be? Does this really serve the meritocracy?
A6: Experience with the Hubble Space Telescope dual-anonymous process indicates that there are few unintended consequences. However, NASA is proactively taking steps to ensure:
- The SMD programs in the pilot lend themselves to dual-anonymous peer review.
- Proposers have sufficient information and guidance to adequately anonymize their proposals.
- Review panels are sufficiently briefed about dual-anonymous peer review.
- The duration of each panel is not significantly increased.
- High-risk/high-impact proposals are not disproportionally affected.
Q7: The DAPR Guidelines used say that an additional page was allotted for the Proposal Summary. But I don’t see that anymore, what happened? ?
A7: This is no longer needed since reviewers will now be able to see the summary from the NSPIRES cover page. So ensure that your summary on NSPIRES is anonymized! .
Q8: Is a table of contents permitted? I cannot find any mention of it in the DAPR guidance or in the text of B.4 HGIO.
A8: The Guidebook for proposers and the ROSES summary of solicitation set the default rules unless they are superseded by, e.g., B.1 The Heliophysics Research Program Overview or the program element. Since both the Guidebook for proposers and the ROSES summary of solicitation allow for a Table of Contents (of up to one page) this is permitted unless it is specifically prohibited. There is no mention of this in either B.1 or in B.4 Heliophysics Guest Investigators Open so it is permitted and it does not count against the page limited S/T/M section.
Q9: Even if the references are done by number [1, 2, etc.] and referencing is done in the 3rd person, there will be good number, maybe even dominant number, of references to my own prior work. So, it is very conceivable that a reviewer could guess by that who is proposing or at least from what group or institution the proposal comes.
A9: Yes, this is unavoidable to some extent. NASA chose square brackets in the text of the anonymized proposal so as not to keep reminding the reviewers who the proposers are likely to be during the reading of the proposal. However, absent a bibliography, the reviewers will have no means to assess any claims made in the proposal. Thus, we opted for a balanced way of handling references. One note: there may be a dominant number of references to a certain author’s work, but the dual-anonymous process still does not make it 100% clear who the PI is. For example, the PI could be a different or former member of the group cited frequently in the proposal.
Q10: Could you provide more information on the content of the budget and justification in the anonymous proposal?
A10: First and foremost, proposers should understand that there has been no change in the requirement that all information pertaining to salary levels, benefits, and overhead rates be redacted from the budget and budget justification included in the main proposal document. This requirement is the same as it has been for the last several proposal cycles. Beyond this requirement, under the dual-anonymous peer review process, proposers should also take care to anonymize the budget and budget narrative in the main proposal document by removing any language, logos, etc. that would reveal the names and/or institutions of the members of the proposing team. Note that neither of these requirements--redaction and anonymization--apply to the "Total Budget" document, which is uploaded separately from the main proposal document and is not seen by the reviewers.
Q11: Should I anonymize the metadata in my proposal?
A11: Yes. Please ensure that metadata (e.g., PDF bookmarks) that could give information about the proposing team and/or institutions is redacted.
Virtual Town Halls on Dual-Anonymous Peer Review
Recording of What’s new in ROSES-2024 and an Overview of Dual-Anonymous Peer Review (YouTube)
Aura Science Team/ACMAP and Earth Science USPI: On June 30, 2022, NASA’s Earth Sciences Division hosted a town hall to discuss the implementation of dual-anonymous peer review for the Aura Science Team/ACMAP and Earth Science USPI programs. The slides from that presentation may be found here.
Cryospheric Sciences: On June 3 and June 15, 2021, NASA’s Earth Science Division hosted virtual town hall meetings to discuss the implementation of dual-anonymous peer review for the Cryospheric Sciences program element. The slides may be found here.
Planetary and Exoplanets Research Programs: On April 28, 2021, NASA’s Planetary Sciences Division hosted a town hall to discuss the implementation of dual-anonymous peer review for Planetary and Exoplanets Research Programs. A recording of the town hall is below and the slides from that presentation may be found here.
Habitable Worlds: On October 7, 2020, NASA’s Planetary Sciences Division hosted a town hall to discuss the implementation of dual-anonymous peer review for the Habitable Worlds program. The slides from that presentation may be found here.
Earth Sciences USPI: On June 3, 2020, NASA’s Earth Sciences Division hosted a town hall to discuss the implementation of dual-anonymous peer review for the USPI program. The slides from that presentation may be found here.
Astrophysics Data Analysis Program (ADAP): There was a special Webinar for Dual-Anonymous Peer Review for the Astrophysics Data Analysis program held on April 2, 2020. The slides from that webinar may be found here.
SMD-wide town hall: Dr. Thomas Zurbuchen, Associate Administrator, NASA’s Science Mission Directorate, hosted a virtual community town hall on March 3, 2020 to discuss the implementation of dual-anonymous peer review in ROSES-20. A recording of the town hall is below and the slides may be found here.
Astrophysics GO/GI: NASA's Astrophysics Division held a virtual community town hall on February 27, 2020 to discuss the implementation of dual-anonymous peer review for Astrophysics GI/GO programs (D.5, D.6, D.9-D.11). The slides from that presentation may be found here.
Questions about specific program elements reviewed under DAPR should be sent to the point of contact in the summary table of key information at the bottom of the program element and posted on the program officer list. General questions regarding DAPR may be directed to Douglas Hudgins douglas.m.hudgins@nasa.gov and cc sara@nasa.gov.
