Monthly growth of preprint review. Data provided by Sciety.

Defining preprint feedback and review

Preprint feedback is publicly available commentary on a preprint that is written by a human.

A preprint review is a type of preprint feedback that has:

• Discussion of the rigor and validity of the research 
• Reviewers’ competing interests declared and/or checked 
• Reviewer identity disclosed and/or verified, for example by an editor or service coordinator, or ORCID login

Calls to action 

1. Individual researchers 
   1. Request reviews and feedback for the next preprint that you post by submitting to a review service (such as Peer Community In, eLife, Review Commons, PeerRef, etc) and/or including an explicit invitation to review on the first page of your preprint or other public location. 
   2. Write preprint reviews following recommended good practices (see FAST principles) and post them as citable objects using a service such as PREreview, Qeios, or ScienceOpen. These may be reviews requested by a journal editor (see Publish Your Reviews) or those you decide to write independently. Consider informing authors about your review ahead of posting and leave them time to provide a thoughtful response. 
   3. Agree to review preprints when invited by platforms such as Peer Community In, Review Commons, Rapid ReviewsInfectious Diseases, and eLife. 
   4. Convert your lab or graduate program journal club to a preprint review club in which discussions are written up, shared with the preprint authors for feedback, and publicly posted.
   5. List preprint reviews on your lab website or via a tool like Sciety to promote their visibility.
2. Funders, departments & institutions
   1. Consider preprints and their reviews in evaluations for funding, hiring, degree requirements, fellowship eligibility, tenure, or promotion. Make this consideration explicit on your website and in application instructions, for example by adopting a CV format that enables listing preprints and their reviews (where candidate is an author of a preprint) and reviews of preprints (where candidate is a preprint reviewer).
   2. Allocate funding and support for preprint review services.
3. Journals
   1. Accept preprint reviews as transferred reviews to inform editorial decisions.
   2. Encourage or require preprint posting at submission.
   3. Partner with preprint review initiatives.
   4. Consider posting reviews on preprints prior to acceptance.
   5. Implement a written policy encouraging preprints and preprint reviews. Suggested text has been recommended by the Journals & Preprint Review Projects Working Group.
   6. Consider adopting a preprint-review model for your journal (e.g., eLife, Peer Community Journal). 
   7. Implement preprint scoop-protection policies (examples: EMBO Press, PLOS, The Company of Biologists) to allow time for preprint review to proceed.
4. Preprint review services
   1. Facilitate preprint reviews meeting the criteria above, investing additional editorial or technical resources into validating identity and addressing competing interests as required.
   2. Create machine-readable metadata for preprint reviews, for example by registering DOIs or providing an API.
5. Preprint servers, indexing & search tools
   1. Link preprints to reviews in a human- and machine-readable fashion.
   2. Partner with preprint review services to allow authors to solicit reviews at the time of submission.
6. Journalists and other non-specialist readers
   1. Seek preprint reviews to provide additional perspectives on research you may cover or use. 

Find the paper on OSF Preprints. We invite everyone to provide feedback and reviews of this paper and to share them publicly.

Download PDF and cite as: Bertozzi, Stefano, Bloom, Theodora, Bourguet, Dennis, Brown, Katherine, Dawson, Stephanie, Edmunds, Scott, Guillemaud, Thomas, Hurst, Phil, Lacy, Michael, LaFlamme, Marcel, Lemberger, Thomas, Lumb, Elliott, MacCallum, Catriona, Pattinson, Damian, Polka, Jessica, Saderi, Daniela, Stell, Brandon, Swaminathan, Sowmya, & Puebla, Iratxe. (2023). Recommendations on Recognizing Preprint Review from the ASAPbio Journals & Preprint Review Projects Working Group. Zenodo. https://doi.org/10.5281/zenodo.7584240

The growth in use of preprints in the life sciences has opened up opportunities for innovation in the evaluation, discussion and review of research works. In the last few years, different projects and communities have started to provide comments and reviews of preprints, experimenting with a variety of models: readers may comment on preprints via the commenting features enabled by some preprint servers, or via platforms that allow commenting on the full preprint or only parts of it (e.g., PREreview, PubPeer), while some projects coordinate the review of preprints via a process similar to traditional journal peer review (e.g., Review Commons, Peer Community In, PeerRef).

Preprint feedback or preprint review?

We recognize that reactions and assessments on preprints can be very diverse in scope and format. For the purpose of these recommendations, the Working Group used the following definitions:

Preprint feedback: Any public commentary on a preprint that adds to scholarship by providing an evaluation of aspects of the study or parts of the manuscript.

Preprint review: A subset of preprint feedback generated via a process that provides transparency on the integrity of the evaluation and includes, at a minimum: 1) an assessment of the rigor and validity of the research and 2) verifiable information (direct or indirect) about the reviewer that allows their expertise and any competing interests to be checked.

Why incorporate preprint reviews into journals’ editorial processes?

Preprint feedback and reviews provide many benefits for all who engage with preprints, journals can also benefit in the following ways from incorporating preprint reviews into their editorial processes:

• Preprint review ‘democratizes’ feedback, allowing any reader to participate in peer review, which can increase diversity and provide a broader pool of potential reviewers for editors
• Greater scrutiny on the manuscript, increasing transparency and providing opportunities to identify issues not covered by the journal’s review
• Potential for quicker editorial decisions once the manuscript is submitted to the journal by using the review(s) on the preprint
• Efficiency via the reuse of the same review by different journals, reducing reviewer burden

All of these outcomes can also be beneficial to authors and reviewers themselves in addition to editors and publishers.

Incorporating preprint feedback and reviews into editorial processes

The Working Group identified four possible journal scenarios where preprint feedback and reviews may be incorporated into editorial processes: informing whether to send a manuscript for review, deciding whether a manuscript is worth highlighting, informing editorial decisions after review (i.e. complementing journal-solicited reviews), and using preprints reviews as transferred reviews within the journal’s process (i.e. replacing journal-solicited reviews). The Working Group then considered the elements needed to enable use of preprint feedback at journals: the table below outlines the elements of preprint feedback required for its use as part of the journal’s process in each scenario.

Preprint feedback elements that may optimize use for different scenarios	Informing whether to send a paper for review	Deciding to highlight a paper (e.g. curation, commissioning articles)	Informing editorial decisions after review	Using preprint reviews as transferred reviews
Public review reports (with date)	✓	✓	✓	✓
Reviewer identity via:- editor role that verifies identity & COIs, or- self-initiated signed review + mechanism for identity verification (e.g. via ORCID)	✓	✓	✓	✓
Assessment of rigor & validity of the research			✓*	✓
Assessment of the novelty of the work, relevance, advance in the field		✓		✓
Author initiated Editor/coordinator role Preprint version Author response Open research outputs Structured assessment				✓
Examples of feedback at preprint-review platforms that fulfill the requirements			(* may vary across reviews on PREreview, the templates encourage an assessment of rigor & validity but this is not a requirement)

Considering the different options available to engage with preprint reviews, there are already examples of journals that are permitting this type of activity or even engaging with it more directly. 

Most journals allow preprints and have no articulated objection to authors receiving feedback or reviews on their preprint. In addition, several journals have partnerships with one or more preprint review platforms to transfer and reuse preprint reviews, examples include the Review Commons affiliate journals, the Peer Community In friendly journals, and the PeerRef partner journals.

Considerations & open questions

While developing these recommendations, the Working Group acknowledged that the approach to preprint review may vary across publishers and journals, and that additional technical tools may be needed to support adoption of preprint review by journals. We discuss some items that are likely to require consideration by journals seeking to engage with preprint review:

Who counts as a peer? 

Views as to who can be designated as a peer, and thus considered qualified to review a paper will vary from one journal to another. Some journals strictly require a depth of expertise proximate to the paper’s topic as part of their reviewer selection, while others seek to include a broader range of perspectives, such as technical reviewers, patients or advocacy groups. Journals may consider whether or not to accept a specific review based on their assessment of the expertise of the person providing feedback and the types of input they typically seek as part of their review process. Such considerations are likely to influence whether journals are willing to broadly engage with preprint feedback, or only with reviews by those who meet their requirements for a peer reviewer.

Self-initiated vs platform-coordinated preprint reviews

We recognize the value of diverse contributions to commentary on preprints, and that platforms that allow individuals to contribute self-initiated feedback facilitate greater diversity in this discourse. At the same time, implementation on the journal side is more likely to take place, at least in the initial stages, in the context of preprint reviews provided by platforms that have a structured process (e.g. that include consideration of reviewer expertise, check for competing interests, or enable an author response).

Barriers to incorporate preprint reviews into journal processes

It can be difficult in the current ecosystem to track all possible comments or reviews associated with a preprint. For preprint reviews to be incorporated into journal processes, locating them must incur minimal additional burden for journal editors, reviewers and authors. Tools that facilitate automated transfer of preprint reviews into journal platforms are needed to enable broader adoption. 

In addition to technical solutions, it will be important to gain a greater understanding of what elements of preprint reviews and the processes behind them can help build trust among journal editors.

Engaging with preprint reviews – a few steps for journals

We suggest a few steps that journals can take to engage with preprint review, and which can be implemented based on existing frameworks and tools:

• Develop and publicly share editorial policies (see policy template below) that articulate the journal’s support for preprint feedback and review, and how the journal uses preprint reviews. 
• Allow authors to share any reviews they have received on their preprint via the journal’s submission process, and surface such reviews to the handling editor. 
• Partner with a preprint-review platform, such as Review Commons, Peer Community In, PREreview or PeerRef, taking into account the frameworks the various platforms operate under and any specific requirements for peer review at the journal.

Appendix: Sample editorial policy text

{Journal name} allows authors to share their research manuscripts on preprint servers, and encourages open communications about preprints between researchers, both on preprint servers and preprint-commenting platforms. When submitting to {Journal name}, authors are welcome to share comments or reviews they have received on their preprint; this information may be provided in the cover letter accompanying the submission {(if applicable) or via the submission form on the manuscript-submission platform}. The comments and/or reviews on the preprint may be used during the editorial process for the manuscript at the editor’s discretion.

(When there is partnership with a preprint-review platform) {Journal name} has a partnership with {preprint-review platform}, enabling authors who had their preprint reviewed by {preprint-review platform} to submit the manuscript and the reviews for consideration. The {Journal name} editors will use the preprint reviews transferred from {preprint-review platform} to inform their editorial decision without restarting the review process.

Working Group Participants

• Stefano Bertozzi, Rapid Reviews: Infectious Diseases, 0000-0002-1723-7085
• Theodora Bloom, BMJ, 0000-0002-0222-4177
• Dennis Bourguet, Peer Community In, 0000-0002-2109-5323
• Katherine Brown, Company of Biologists, 0000-0001-9110-8276
• Stephanie Dawson, ScienceOpen, 0000-0002-2298-2418
• Scott Edmunds, GigaScience, 0000-0001-6444-1436
• Thomas Guillemaud, Peer Community In, 0000-0003-0451-1644
• Phil Hurst, Royal Society ,0000-0001-7971-1005
• Michael Lacy, ASCB MBoC Preprint Highlights, 0000-0003-0498-2817
• Marcel LaFlamme, PLOS, 0000-0002-7489-4233
• Thomas Lemberger, EMBO, 0000-0002-2499-4025
• Elliott Lumb, PeerRef, 0000-0001-8703-6097
• Catriona MacCallum, Hindawi, 0000-0001-9623-2225
• Damian Pattinson, eLife, 0000-0002-2606-6599
• Jessica Polka, ASAPbio, 0000-0001-6610-9293 
• Daniela Saderi, PREreview, 0000-0002-6109-0367
• Brandon Stell, PubPeer, 0000-0003-1287-9211
• Sowmya Swaminathan, Springer Nature, 0000-0002-0285-4910
• Iratxe Puebla, ASAPbio, 0000-0003-1258-0746 

Download PDF and cite as: Avissar-Whiting, Michele, Belliard, Frederique, Dumanis, Sonya, Eldon Whylly, Kristin, Farley, Ashley, Franko, Maryrose, Hazlett, Haley, Neves, Kleber, Puebla, Iratxe, Rooryck, Johan, Royle, Stephen, Spiro, John, Stern, Bodo, Strinzel, Michaela, & Polka, Jessica. (2023). Recommendations on Recognizing Preprint Review from the ASAPbio Funder, Researcher, and Institution Working Group. Zenodo. https://doi.org/10.5281/zenodo.7584319

The broader use of preprints has led to a wide range of activities involving reactions, comments and review of preprints. This means that in the current ecosystem, peer review also happens outside of journals and can take place on research outputs shared as preprints, facilitating a more open and diverse discourse on research works. As this type of activity flourishes, it is important for funders and institutions to consider those forms of review as valid scholarship within their research assessment processes.

Why incorporate preprint reviews into funder and institutional research assessment processes?

The traditional journal publication framework imposes limits on the types of research outputs considered valid for evaluation and funding decisions by funders and institutions, and it has created known inequalities. The digital publishing ecosystem now has the tools to not only allow researchers to share a wide range of research outputs, but also to decouple publication from review. It is thus relevant to update research assessment frameworks to include a broader view of a researcher’s contributions. Preprint review provides the following benefits to funders and institutions:

• Opportunities for faster scientific scrutiny and corrections – science can be assessed and corrected faster, increasing failure efficiency and avoiding redundancy or work on research avenues that are not productive
• Catalyzes culture change, normalizing open participation in scientific dialogue 
• Signals the value of reviews, both as scholarly contributions in their own right and as further context that adds to the understanding of the paper by different audiences
• Makes better use of researcher time and intellectual energy, as reviews are not used only as a curation mechanism but are preserved as part of the scientific dialogue
• Provides an opportunity to address bias in the evaluation and non-collegial feedback
• Increases transparency in review and opens up the scholarly communication process

Working definition for preprint review

While there can be many types of feedback on a preprint (ranging from brief comments on social media to full in-depth reviews), for the purpose of these recommendations, the Working Group used the following definition:

A preprint review is an assessment of a preprint that includes:

• Rigor and validity of the research discussed
• Consideration of the reviewer’s competing interests, either through a check by an editor/coordinator, if available, or via a public declaration of competing interests by the reviewer
• Mechanism to confirm the reviewer identity, either via verification by an editor/coordinator, if available, or via a public declaration by the reviewer (i.e. the reviewer signs the review) along with ORCID verification

A preprint with reviews is a preprint that has two or more reviews as defined above. Table 1 lists examples of practices by services and/or platforms that provide these reviews, including several that provide reviews fulfilling all of the criteria outlined.

Table 1: Practices of example preprint review services and/or platforms

	F1000 Research example	eLife example	Review Commons example	PCI example	PREreview example	preLights example	Preprint server comment example
Rigor/validity addressed						possibly	possibly
Reviewer competing interests – checked by editor or publicly declared	declared	checked	checked	checked	declared	Not included by default	Not included by default
Reviewer identity – verified or signed (w/ORCID notification)	verified & ORCID	verified & ORCID	verified	verified	Can be signed, but no ORCID notification	verified	Can be signed, but no ORCID notification

Incorporating preprint feedback and reviews into research assessment processes

Many funders have policies that recognize preprints as valid research outputs for the purposes of grant applications and reports (asapbio.org/funder-policies), and many institutions also encourage preprints as part of hiring and promotion (asapbio.org/university-policies). While the use of preprint reviews is less common, there are some examples of funders and institutions that already endorse the use of preprints with reviews and preprint reviews in research assessment. As noted in Table 2, some organizations have already signaled that preprints with reviews fulfill their requirements for peer-reviewed publications.

Table 2: Factors in Preprint Review Recognition

Funders and institutions seeking to incorporate preprints with reviews or preprint reviews into their processes can consider the following areas and policies for implementation:

• Requirements for PhD graduation or postdoctoral fellowships: Include refereed preprints as fulfilling graduation or fellowship eligibility requirements where a peer-reviewed publication is required. As an example, EMBO’s Postdoctoral Fellowships require that candidates have a first-author peer-reviewed article (which may be a refereed preprint), and graduate programs in France accept preprints recommended by Peer Community In as fulfilling graduation criteria.
• Grant applications and reports, hiring, promotion: Indicate as part of guidelines for grant applications and reports, and in job applications, that preprints with reviews may be included, e.g. cOAlition S organizations have indicated that they consider preprints with reviews as equivalent to peer-reviewed articles published in a journal, and 9 funders and research organizations supporting eLife’s editorial model have affirmed they will consider reviewed preprints in their evaluation processes. As part of the bibliography of candidates’ work, include sections where applicants can highlight open peer reviews and endorsements of their preprints, as well their own output of public reviews.
• Recruitment of grant reviewers: When recruiting reviewers for grant applications, assess the researcher’s output of public reviews (via their ORCiD, CV, etc) to identify suitable reviewers. 
• Reporting, monitoring/evaluating program impact: Include preprints with reviews and public review output when evaluating a department/unit/program’s research output.

Challenges and limitations

While the Working Group believes there are many benefits to preprint review, we also acknowledged potential considerations and challenges around the implementation of preprint review in research assessment:

• Place value on open scholarly discourse, rather than using preprint review as a proxy for quality: Recognition for preprint review must be based on assigning value to openness when engaging on scholarly discourse around research works. We should nurture a culture of openness and of engagement with public discourse, while recognizing that the assessment within reviews does not necessarily determine the quality of the preprint.
• Peer review is not incentivized: Many academics are already overwhelmed with requests to review from journals, which itself carries few concrete benefits. This makes it challenging to devote additional hours to new forms of reviewing.
• Change is slow: Scholarly publishing, and academic culture in general, can be slow to change. Policies and initiatives must work to evolve entrenched practices.
• Undesirable hierarchies: Recognition for preprint review should not create an artificial hierarchy in which preprints with reviews are valued above those without; it will be important to preserve the value of a standalone preprint as a research output, rather than ratcheting expectations that they should all be reviewed.
• Recognition for preprint review must build on recognition for preprints: Preprint review is layered on preprints and thus requires a robust pool of preprints to evaluate. It is thus necessary to support further recognition of preprints themselves as part of research assessment.
• Scaling: While there are different preprint-review initiatives, they are currently operating at a relatively small scale. To increase the visibility and value of preprint reviews for researchers, it is important to scale up preprint review platforms and/or services, so that they can meet demand as this grows.
• Infrastructure: Preprint reviews currently exist across a wide range of platforms, and at the same time readers are unlikely to seek out reviews independently of their respective preprints. It will be important to continue to develop infrastructure that facilitates linking and connections across the distributed network of preprint reviews and improves their visibility in association with individual preprints and related scholarly outputs.
• Most reviews are still closed: The majority of peer review still takes place at journals within a closed/confidential system available only to a select few, thus keeping reviews inaccessible to the broader community. It will be important to support transparency in review not only on preprints but also for journal articles.

Working group participants

• Michele Avissar-Whiting, HHMI, 0000-0003-0030-3135
• Frederique Belliard, TU Delft, 0000-0001-8750-7207
• Sonya Dumanis, Aligning Science Across Parkinsons, 0000-0002-3345-9497
• Kristin Eldon Whylly, Templeton World Charity Foundation, 0000-0002-4677-148X
• Ashley Farley, Bill & Melinda Gates Foundation, 0000-0001-9310-6944 
• Maryrose Franko, HRA, 0000-0003-1870-8472
• Haley Hazlett, DORA, 0000-0001-9937-435X
• Kleber Neves, Serrapilheira Institute, 0000-0001-9519-4909
• Iratxe Puebla, ASAPbio, 0000-0003-1258-0746 
• Johan Rooryck, cOAlition S, 0000-0001-7214-7405
• Steve Royle, Warwick, 0000-0001-8927-6967
• John Spiro, Simons Foundation, 0000-0003-2820-5844
• Bodo Stern, HHMI, 0000-0002-4900-936X
• Michaela Strinzel, SNSF, 0000-0003-3181-0623
• Jessica Polka, ASAPbio, 0000-0001-6610-9293 

The survey garnered 230 responses, and we share here summaries of the two largest categories of respondents: 161 responses from researchers and 51 responses from journal editors and publishing organization employees. You can view the results on Google Sheets and on Zenodo.

Researchers would be incentivized to participate in preprint review by funder & journal recognition

Most researcher respondents were group leaders, and ⅔ described their primary discipline as biology. Our sample displays a strong geographic bias toward North America and Europe, with only 20% of respondents coming from outside of these continents.

Most respondents had received no feedback on preprints, which, for the purpose of this survey, we defined as any public commentary on preprints. Of those who had received some feedback, only a small fraction indicated that the feedback came in the form of detailed reviews. 

With few researchers having received feedback, perhaps it’s unsurprising that a significant number of them expressed concerns with the prospect: the most significant concerns related to hesitancy about the quality or fairness of the feedback and about the commenter’s motivations for providing it.

However, more than half of respondents said they’d be likely or very likely to request feedback on their preprints if journals incorporated preprint reviews into editorial decisions or treated them like reviews transferred from another journal. Other potential incentives, such as funders recognizing preprint peer reviews in various ways, were not far behind.

Similar to their (sparse) experience receiving public feedback, relatively few researchers had posted comments or reviews on preprints themselves. The most common reason given for not engaging was “lack of time,” which we speculate means that researchers see providing preprint feedback as an activity that may yield small payoffs in terms of recognition or utility to others, compared to other research contributions. As above, many researchers indicated that formal recognition for preprint review by both journals and funders would encourage them to participate.

Journal editors and publishing organization employees cite trust and discovery as key challenges; favor reviews created by organized platforms

Among journal editors and publishing organization employees who took our survey, many identified as in-house or academic editors. 60% were from biology journals with the majority based in Europe.

Nearly half of this group reported that their journal had previously incorporated preprint reviews produced by a platform such as Peer Community In or Review Commons. Given that the number of journals offering these integrations is small, our sample represents community members at journals who are relatively progressive with regards to incorporating preprint feedback. Indeed, many respondents indicated that they were motivated to engage with preprint feedback for a variety of reasons, most commonly to check for potential concerns with the paper that should be considered, reuse feedback to streamline their processes, and identify new reviewers.

Regarding challenges to engage with preprint feedback, more than half of respondents reported being discouraged by the small number of preprints with feedback, the difficulty of finding that feedback across different platforms, and doubts about feedback integrity or commenter motivations.

Given the challenges with discovery and trust, it’s unsurprising that many more respondents said their journal would be willing to reuse preprint reviews that originated from a platform that organizes and oversees the peer review process versus reviews posted independently by individual reviewers. The majority of respondents said that this coordination by a platform was important or very important for enabling their journal to reuse preprint reviews. However, even more indicated that a mechanism to verify the reviewer’s identity was a key feature of the review. 

In the coming weeks, we’ll share additional outputs of the Recognizing Preprint Peer Review workshop that delve deeper into strategies for different stakeholders to support preprint review and tackle these key emerging challenges of recognition and trust.