Preprint reviews hold the potential to build trust in preprints and drive innovation in peer review. However, the variety of platforms available to contribute comments and reviews on preprints means that it can be difficult for readers to gain a clear picture of the process that led to the reviews linked to a particular preprint. 

To address this, ASAPbio organized a working group to develop a set of features that could describe preprint review processes in a way that is simple to implement. We are proud to share Preprint Review Features (PReF) in an OSF Preprint. PReF consists of 8 key-value pairs, describing the key elements of preprint review. The white paper includes detailed definitions for each feature, an implementation guide, and an overview of how the characteristics of active preprint review projects map to PReF. We also developed a set of graphic icons (below) that we encourage the preprint review community to reuse alongside PReF. 

While the Peer Review Terminology developed by the STM working group and the Open Peer Review taxonomy provided useful background for our discussions, they were designed with a focus on journal-based peer review, and do not capture all the possible elements that can be part of preprint review. We acknowledge that there are nuances and different views as to what constitutes “peer review,” “feedback,” and “commenting;” rather than create strict definitions, our aim was to parse out important aspects of the process involved in any form of review on preprints, and to do so in a format that could be used by platforms that host, coordinate, or aggregate such activities. Therefore, we are glad to see that PReF is already implemented on ReimagineReview and on review aggregators like Early Evidence Base and Sciety. We hope that our efforts in the development and adoption of PReF will promote better visibility and discoverability of preprint review.

On ReimagineReview, PReF can be used as a search filter for preprint review services. For example, you can select “Active” projects with “Preprints” as the type of output reviewed, using PReF as filters, you can identify preprint review services where the Authors request review and Author response is included. 

On Early Evidence Base, you can find preprints linked to preprint review with a description of the review service using PReF. Early Evidence Base aggregates referred preprints from Review Commons, Peerage of Science, PeerRef, EMBO Press, eLife, Peer Community In, and Rapid Reviews COVID19.

Sciety is a platform where researchers and groups can curate and find peer-reviewed preprints. Sciety uses PReF to describe the review processes of Sciety Groups. 

Continued tracking of preprint review trends

We provide a simple analysis of the current active preprint review services listed on ReimagineReview as of December 2021. Please see the appendix of the White paper to read about the current landscape.

We encourage the continued tracking of preprint review projects in the future to identify emerging trends. ReimagineReview data will be available on Zenodo quarterly. Please find the data on preprint review projects with the PReF fields here, and the code used to generate the preliminary analysis is linked here. 

Evolving terms for an evolving field

PReF is one of the first attempts to describe the processes behind preprint review; we expect continued innovation in the preprint review field and thus potential improvements to PReF. Regardless of the direction of evolution, we believe clear and consistent descriptions of preprint review processes will foster transparency. We look forward to continued conversations around the features behind peer review processes across a broad range of research outputs including preprints. 

Working group members (listed alphabetically)

• Michele Avissar-Whiting (Research Square)
• Philip N. Cohen (SocArxiv)
• Phil Hurst (Royal Society)
• Thomas Lemberger (EMBO/EEB/Review Commons)
• Gary McDowell (Lightoller/Doc Maps)
• Damian Pattinson (eLife)
• Jessica Polka (ASAPbio)
• Iratxe Puebla (ASAPbio)
• Tony Ross-Hellauer (TU Graz/Doc Maps)
• Richard Sever (Cold Spring Harbor Laboratory and bioRxiv/medRxiv)
• Kathleen Shearer (COAR Notify)
• Gabe Stein (KFG/Doc Maps)
• Clare Stone (SSRN)
• Victoria T. Yan (ASAPbio, presently EMBL)

Huge thanks to everyone in the Working Group. This work was the result of many stimulating discussions with a very diverse group. 

We are glad to see PReF in use. To provide feedback on our work, or if you have questions about implementing PReF, please leave a comment here on this blogpost, on the white paper, or email Victoria @ victoria.yan@embl.de. 

​​

While preprints have been adopted as a means to promptly disseminate research, they also open up new ways to participate in the scientific discourse around the latest research. There are many benefits to public feedback on preprints: comments that can help authors improve their work, broader opportunities for early career researchers to participate in review, and additional context for readers. However, we have not yet seen wide engagement in the public review of preprints. This is likely due to cultural barriers: there is a lack of incentives for researchers to participate in preprint review, but there can be risks associated with posting pointed critiques, however constructive, on the paper by another researcher whose favour you may require for a future job or grant.

If we are to foster a positive and thriving environment for public preprint feedback, we need to collectively agree on the norms and behaviors we expect when creating, responding to, and interpreting preprint feedback. This was the remit of the ASAPbio preprint review cultural norms Working Group, which has been developing a set of principles for preprint feedback over the last six months. Following the initial draft shared last July, the Working Group has iterated on the principles based on feedback and has discussed their potential use by different stakeholders in science communication.

We are pleased to now share the FAST principles for preprint feedback. This is a set of 14 principles clustered around four broad themes: Focused, Appropriate, Specific, and Transparent (FAST). Each principle includes a designation for the actors it applies to: authors, reviewers and the community. 

To develop the principles, the Working Group reviewed existing resources on journal peer review, preprint review and public commenting on scholarly work and discussed how those align or differ with preprint feedback and the set of behaviors and cultural norms we would like to see as a community. While some principles should remain the same whether you are reviewing a manuscript for a journal or leaving a comment on a preprint, there are some particularities about preprints (open to everyone, not geared to a certain journal, etc) that require a specific set of principles. For example, (most) journal peer review is anonymous and reviewers are selected by a journal editor based on their expertise; preprint commenting, on the other hand, is open for everyone. In the context of preprint feedback, we could in principle promote a culture where we encourage signing the feedback if the commenter is comfortable doing so, and if they are not, recommend that they specify their area of expertise when commenting about certain aspects of a preprint. Importantly, we have also considered that in an environment of preprint feedback, there is a broader set of stakeholders who can interact and reply to preprint comments. As a result, there are also expectations for authors and readers to display a set of behaviours that enhance and promote a healthy, valuable and productive culture for public preprint feedback.

The FAST principles constitute a first attempt to generate cultural norms around preprint feedback. However, we do recognize that we are at early stages of preprint feedback activities and we expect (and hope) that these principles will constitute an initial framework for different stakeholders to use, incorporate and build on. For example, the FAST principles can be used by researchers when providing feedback in the context of institutional or cross-institutional journal clubs and similar initiatives. Moreover, preprint-review platforms could mark reviews as FAST-compliant and this could be particularly helpful for journals when considering whether to bring preprint reviews into their editorial process. There are additional potential uses for the FAST principles, and we are certain that there may be applications that we have not yet envisioned and which will arise as the ecosystem develops.

Here you can find more information about the FAST principles, including examples of preprint feedback statements that reflect each of the principles and the resources that the Working Group used. We would like to thank all those who contributed to the principles, including those who discussed them with us at the FeedbackASAP session and those who reviewed different iterations of the principles.

The FAST principles constitute a set of initial recommendations for the different actors that participate in scholarly communication so we can all agree on what are the best practices when providing preprint feedback. We encourage all stakeholders to incorporate the FAST principles into their existing routines and to provide feedback regarding the principles or issues that might arise with their implementation. Please comment on this post, on the preprint or send an email to Iratxe Puebla. Your feedback will help us adapt the FAST principles to continue to support a thriving preprint feedback culture.

Finding relevant preprints

Christine Ferguson and Martin Fenner outlined their proposal to develop ways for researchers to find preprints relevant to their research immediately after the preprints appear. They propose an automated system that would identify preprints posted in the previous few days that had received attention via Twitter (i.e. based on the preprint receiving a minimal number of tweets).

During the discussion, the session attendees mentioned a number of currently-available tools that collect reactions and attention on preprints and/or allow researchers to discover the latest preprints:

• CrossRef collects Event Data for individual preprints, including social media mentions, Hypothes.is annotations and more.
• The Rxivist.org tool allows searching for bioRxiv and medRxiv preprints based on Twitter activity.
• The search.bioPreprint tool developed by the University of PIttsburgh Medical Library allows searching preprints from different servers based on keywords or topics.
• bioRxiv provides search options based on discipline and also has a dashboard that collects reactions and reviews on individual preprints, including Twitter comments.
• EMBO has developed the Early Evidence Base platform which allows searching for refereed preprints. 
• Google Scholar indexes preprints and provides some filtering tools.

The attendees raised some questions about the use of Twitter as a filter and the risks for a metric based on tweets. How can we account for the risk of social media users gaming the system by artificially boosting attention on Twitter? How can we normalize for the fact that methods papers tend to receive more attention? Is there a risk that this system will be focused on papers from high-income countries that already receive a disproportionate share of attention?

Christine and Martin noted that their proposal involves a direct weekly email to researchers as the channel for communication, allowing those who sign up to relevant subject areas to receive a list of preprints without them having to individually mine different tools. Their proposal does not sort or rank papers by the number of tweets, but rather requires a minimum threshold of Twitter activity for inclusion. They recognized that there may be a number of biases associated with attention to preprints, some we are aware of and some which we still may not know about, but they propose their approach as a starting point which can be developed further.

The group suggested setting up a form to which researchers could sign up to signal interest in receiving the proposed newsletters. To this end, those interested in using and feeding back on the early prototype, can sign up via this link: https://front-matter.io/newsletter, and will receive weekly lists of preprints in specific subject areas.  The feedback will be used to fine-tune the newsletters going forwards.

Finding reviewers

In the last part of the session, Daniel Mietchen outlined the differences between preprint and journal review and how some of the advantages of preprint review can be leveraged to find reviewers.

Daniel proposed some recommendations for how preprint servers could attract preprint reviews:

• Use open licenses for preprints and formats that allow reuse.
• Provide mechanisms to flag which aspects of a preprint are in most need of a review – this could be done by authors themselves or by readers who indicate whether e.g. the statistics or a specific methodology should be evaluated by another expert.
• Make it easy to annotate preprints by having HTML versions and enabling Hypothes.is.
• Use non-tracking tools for commenting.
• Make it easy to subscribe to content alerts using specific filters, e.g. based on mentions to specific concepts (cell line, model organism, author, institution) in sections of the paper or to lists of preprints/papers with high similarity to a given piece of text (similar to the JANE tool to search PubMed).

Christine (ferguson.cav@gmail.com), Martin (martin@front-matter.io) and Daniel welcome comments and feedback on their proposals, feel free to comment on this blog post or contact them directly.

At the ASAPbio #FeedbackASAP meeting held on July 21st, 2021, the Sciety team (Hannah Drury, Godwyns Onwuchekwa, and Paul Shannon) led an interactive session examining different aspects of the evolving landscape of preprint curation and review. 

What is curation?

We began the session by brainstorming what curation encompasses. Through a mind-mapping exercise, the group identified highlighting, providing context, and selecting articles as some examples of what curation means. Just like curation that occurs in museums and in art exhibits, curation in research articles plays an important role in preservation as well as providing background context for research articles. 

Not only can the reader learn about articles they might not otherwise have found, but the work is also connected to other relevant papers by virtue of inclusion in a collection that may cover, for example, a specific research topic. While many curation efforts focus on new papers, we discussed the under-recognized benefits of curating past literature rich in insights. However, despite these benefits, curation in certain communities can act as a way of gate-keeping. Counteracting this behavior requires specific attention to our inherent biases and could be addressed by diversifying the curator pool, an approach which Sciety is contributing to achieve.

What differentiates curation from review?

Next, the group was asked to address what differentiates curation and review. Curation and review are intertwined in traditional journal peer review where they occur prior to public dissemination. However, in the case of preprints, curation and review can be uncoupled. The group discussed different points at which these processes could occur in the preprint workflow. Curation is an activity that could be performed at any stage once the author releases the paper into any form of public platform, as well as on finished work. What this brings is the opportunity to make the work a living document and thus can add value.

Another distinction between curation and review is that review is scholarly work that adds additional text alongside the original article rather than sorting the original article into a list. Review acts to evaluate the strength of logic and the supporting evidence. The review reports on the validity of the research and should be performed by experts. In contrast, a wider community could engage in curation.

How can curation be useful?

The group then discussed how curation activity could be most useful. Curation can link related articles to one another and surface them to readers. Curation will be increasingly essential, as there are now 100,000 articles produced per month in the biomedical sciences alone, and especially as we transition to posting research first as preprints. Curation efforts will help researchers save time in navigating relevant articles and potentially amplify unheard voices. This can be powerful for reviewers and authors who are under-represented in science.

Would you like to curate the preprints you are reading? Then, Sciety is an open platform where you can do that with very little effort. Just log, search and start saving your articles of interest, and feel free to share (curate) your page publicly. 

For more detail about the session, check out the post by Sciety did on their blog.

Photo by Jason Leung on Unsplash

Peer reviewing: for whom and why?

The organizers opened the session with some prompts to gauge participants’ experiences with preprinting and peer review. In general, attendees were positive about posting preprints and the experience of receiving feedback on them, but were more ambivalent about performing review on behalf of journals. While participants enjoyed providing feedback to authors, some expressed philosophical discomfort with acting as a gatekeeper. This prompted a discussion, led by Mike Eisen (eLife), about the many roles of journal peer review, one of which is to serve “never readers:” decision-makers and evaluators who will eventually care that the paper was accepted by the journal in question, but never actually read it. Preprint review, instead, serves authors and readers more directly. Free of the need to act as a gatekeeper, reviewers of preprints can write collegial and helpful reviews they’d be happy to share publicly.

Despite the fact that many people are dissatisfied with journal review, few people are reviewing preprints, perhaps because, as one participant suggested, there are no concrete rewards or consequences for doing so. This session aimed to use the existing practice of journal review to accelerate preprint review, creating a more constructive review culture along the way. 

Ludo noted that changes in publication and peer review practices are often slow, perhaps because modifying the system requires the coordination of many different stakeholders: editors, editorial boards, publishers, reviewers, authors, and so on. Arguing that we can speed up innovation via grassroots action that requires little coordination, he proposed to launch a campaign following the lead of session co-organizer and ASAPbio Vice President James Fraser (UCSF).

James Fraser’s preprint reviewing policies

James explained that he has adopted a policy to only accept journal review invitations when the manuscript in question has been posted as preprint; he then posts the review completed for the journal as a comment on the preprint. If there is no preprint, James refuses the invitation to review, citing his policy, and cc’s authors. This strategy has prompted at least one group of coauthors to post their work as a preprint. In order to facilitate this action, both James Fraser and Mike Eisen indicated they’d be willing to help colleagues maintain their anonymity by posting reviews on their behalf.

James has also taken this idea one step further: when asked to guest edit a paper for a journal, he instead organized an independent review process, inviting his own reviewers. With the consent of all reviewers, he then posted the entire package of reviews as a comment on bioRxiv. 

Proposals for campaigns

Ludo set forth a few proposals for pledges or campaigns. The discussion focused on two pledges:

1. As a reviewer for a journal, when I review an article that is already available as a preprint, I will post my review publicly
2. As a reviewer for a journal, I will only review an article if it is already available as a preprint, and I will post my review publicly

Participants raised concerns about how early career researchers (ECRs) might be disproportionately affected by taking these actions: ECRs may be hesitant to post reviews on preprints for fear of offending either authors or journal editors. A number of solutions to this were discussed, for example posting reviews anonymously, pseudoanonymously, or under the name of more established researchers who are less prone to these concerns (e.g., James or Michael). Despite these concerns, many session participants indicated a willingness to sign on to one or both of these campaigns, and organizers have begun developing these ideas into a collective action campaign that will be posted on the Free Our Knowledge website in the coming weeks. In the meantime, interested members of the community are invited to provide feedback and indicate their preliminary interest in the campaign, either by posting comments on the GitHub thread directly or contacting Cooper Smout and Ludo Waltman.

At the July 21, 2021 #FeedbackASAP meeting, Mugdha Sathe (UW), Rebeccah Lijek (Mount Holyoke), Daniela Saderi (PREreview) organized a breakout session on using public preprint review in teaching and mentorship of early career researchers. 

Who is a “peer?”

The session began with Daniela Saderi leading us in discussion about who can be a peer reviewer, challenging the traditional ideas of what we mean by “expertise.” After lively conversation, we came to a crowdsourced definition that a reviewer is “anyone participating in science willing to think deliberatively, critically, and constructively about the work” and that the act of doing peer review (preprint or journal review) is what creates peer reviewers, not a faculty title. 

Preprint reviews fill a gap in science education 

Then we reflected on how we were trained in peer review, broadly agreeing that we hadn’t received formal training, but instead got passive exposure to peer reviews by reading reports on our papers or by producing them ourselves. Becki Lijek presented data corroborating these experiences from her recent survey, which showed that peer review training is rare. Moreover, the lack of training drives ECRs to ghostwrite reviews on behalf of their PIs. For example, 70% of respondents who had coreviewed with their PI made a major contribution, but to their knowledge, were not named to the journal.

She called for a paradigm shift to integrate authentic experiences in peer review—like preprint reviews—into science education. Involving ECRs in preprint review improves their scholarly skills of critical thinking and writing, and it also broadens students’ understanding of science to include community engagement. Since preprints and their peer reviews play a major role in new developments related to COVID-19, understanding of these processes among non-specialists has become even more crucial. 

Preprint review is a meaningful component of the scientific process, and as such, Lijek is studying whether participating in it improves ECRs’ sense of belonging in science. She posits that preprint review provides an exciting new avenue to support the persistence of BIPOC, women, and gender minorities in STEM. 

Mugdha Sathe shared her experience in teaching peer review to undergraduates as the final project in a seminar course for senior biology majors at a large research university. Learning was assessed with a writing assignment, including by asking students to predict the next set of experiments, offer alternative hypotheses, and identify issues with experimental design. Mugdha shared tips such as her use of the tool Perusall for collaborative reading and the rubrics she employed for evaluating students’ reviews.

Creating socially-conscious reviewers 

PREreview Executive Director Daniela Saderi provided a summary of resources for preprint review available through PREreview, including their Open Reviewers community, which she will be working with Becki Lijek to use with undergraduates. Saderi then took us through an exercise in challenging implicit biases and structural inequities in peer review, including a discussion of ageism. In all, we agreed that all ages and career stages have an important role to play in peer review and preprint review opens up new, exciting opportunities for ECRs to learn and engage with the scientific community. 

View collaborative notes, containing links to session slides.