Metadata is structured information that describes research articles; such as the author, title, publication date, keywords and preprint server hosting the article. In scientific publishing, metadata is essential for enhancing the discoverability, accessibility, and reproducibility of research. It allows researchers, databases, and indexing services to efficiently organize, search, and cite scientific work, thereby supporting transparency and collaboration across disciplines.

Ensuring transparency and accessibility for preprint review metadata is paramount. A shared understanding and consistent use of preprint review metadata is crucial, because it can be used to give readers insight into the rigor and thoroughness of the review process. This understanding can impact readers’ perceptions of a preprint’s reliability and validity. In a world where preprinting is the norm, it will be increasingly important to help communicate the value to readers.

This comprehensive resource isn’t just another website; it’s a living guide providing clear examples of metadata associated with preprint peer review. This website provides examples of how and where preprint review metadata is currently deposited and how to retrieve and share it. It is organised as fields that are inputs to or outputs from the process.

Each metadata element is assigned a colour-code indicating how well the information is currently being shared across different preprint peer-review initiatives and where it can be recorded. These scores are visually represented using a clear traffic light system:

Red: Indicates that information is not well captured or implemented, signalling an immediate need for community action

Amber (Yellow): Suggests that data is partially collected or implemented, requiring observation to determine if further community action is needed

Green: Shows that information is well collected and implemented, indicating robust practices

Grey: Signifies that no further action is required at this stage

A Community-Driven Initiative

This page isn’t meant to be a static resource. Each page includes an “actions/updates section” where members of the community can actively discuss steps to improve the deposit and retrieval of metadata. We strongly encourage contributors to make edits and contribute to the ongoing improvement of this vital resource. By providing these insights and tools, the Preprint Review Metadata Modelling documentation aims to foster greater consistency, discoverability, and transparency in preprint peer review. It’s a collaborative effort to ensure that valuable review information is as open and accessible as the preprints themselves. 

Dive In and Contribute! Whether you’re a researcher, a preprint server operator, a funder, or simply interested in open science, we invite you to explore this documentation. Your insights and contributions are invaluable as we work together to enhance the landscape of scholarly communication. If you have any specific questions relating to this resource please contact jonny.coates@asapbio.org

It’s peer review week and to celebrate, we are releasing a preview of one of the 2024 ASAPbio Fellows projects focused on investigating the ASAPbio Crowd Preprint Review activities.

ASAPbio Crowd Preprint Review

The theme of peer review week this year is “Innovation and Technology in Peer Review”. Since 2021, ASAPbio has been running its own version of crowd preprint review. This approach involves using a crowd of reviewers to collaboratively review a manuscript and was developed by the journal Synlett in 2017. The use of a crowd is the innovative aspect to this approach of performing peer review as it enables real-time discussion amongst reviewers and allows individual reviewers to only comment on aspects that fall within their expertise. This means that a large number of small comments can be combined into a traditional-looking peer review. Additionally, ASAPbio posts all reviews publicly which are subsequently linked back to the preprint, providing transparency and greater context for readers.

The ASAPbio approach involves crowd leads choosing preprints and disseminating them to the crowd members for review. Once crowd members add their comments, crowd leads synthesise the comments and discussions into a traditional-looking peer review. They then upload the synthesised reviews so that the reviews are publicly available (Fig 1).

ASAPbio began with a pilot in 2021 focused on cell biology preprints. The following year this expanded to three disciplines, including one in collaboration with SciELO that covered Portuguese-language preprints. In 2023, there were 4 crowds covering preprints across bioRxiv. Four crowds were again launched in 2024 and will run for an entire 12 month period, in contrast to the more tightly defined periods of previous years. Between 2021 and September 2024, this initiative has produced 97 public reviews on preprints (Table 1).

Experience of crowd members

A group of Fellows has been conducting a survey of current and past crowd members to reflect on the experience of being a crowd member. As a preview of our work, we present some data on the survey respondents and the reasons that they joined the program.

The majority of respondents indicated that they had a background in biology or meta-research with remaining disciplines being collated into “other”. Given the historical roots of ASAPbio as a life science organisation, it is unsurprising that biology is so well represented. There has also been a focus on meta-research as a crowd which explains the number of respondents who identified this as being their scientific area of interest (Fig 2A). In terms of career stages, most respondents were post-doctoral researchers (post-docs) or Principle Investigators (PIs) (Fig 2B). There were also 7 students who responded to the survey. Of those who stated that they had never participated in a crowd review, the majority were students or PIs whereas those most actively engaged were the post-docs. Those who identified as “other” job titles were also actively participating.

Why do people participate (or not) in crowd review?

Respondents indicated varied reasons for why they participate in crowd review activities (Fig 2C). Engaging in Open Science practices was most cited, followed by developing peer review skills, and contributing to the science community. Less frequently, respondents mentioned participating in crowd reviews for personal gains, such as increasing their visibility, showing evidence of their productivity, and keeping up-to-date with the latest research in their field. The top reasons for participating collectively represent 3 levels of impact: 1) Society, 2) Self, and 3) The scientific community. This strong emphasis on contributing to open science and self-directed training highlights the two key benefits of crowd preprint review. Indeed, the collaborative nature of the review process was highlighted in several free-text responses, with one responder communicating how it enables “sharing reviewers concerns, coming to an agreement, [and] sharing the workload.”

The two biggest reasons that respondents stated for not reviewing a given preprint were a lack of relevant expertise and a lack of time (Fig 2D). Given the size of the crowds, it is not unexpected that some will not have the specific expertise for any component of a given preprint. However, the benefit of a crowd is that this “expertise” burden is reduced as reviewers do not need to comment on the entire preprint. It is possible that this aspect could be better communicated to encourage people to review more granularly and to accept that as a valid contribution. Academics are under heavy workloads and are time-poor. Additionally, we have noticed in previous iterations of the crowd review program that in the busier months in the academic calendar, there is an anecdotal reduction in reviewing activities. A lack of interest in the topic was the third most common reason for not reviewing a given preprint. Thus far, crowds have been designed to cover entire fields (such as “cell biology” or “immunology”), so it cannot be expected that every chosen preprint will be of interest to the entire crowd.

This preview of our on-going efforts highlights altruism and a desire to participate in better practices as the primary motivator for participating in the crowd review program.

Conclusion

We’re still performing a full data analysis and expect to post a full preprint before the end of the year! But if you would like to join the crowd preprint review efforts for 2024-2025 you can sign up by clicking this link.

I was tired of hearing students and colleagues sharing their bad experiences with reviewers. I was tired of working really hard to submit a paper for publication and receiving inconsiderate reviews and editorial decisions. We needed a change of culture in peer review: change it from being just a “Where’s Waldo” game of pointing out flaws and what the paper didn’t do towards fair criticisms based on what’s presented, and also reinforcing the strengths of the work. I believe that peer review can be kind and fair, while still being rigorous.

As with any cultural change, we need to focus on the new generation, the leaders to come, rather than the leaders in place. The new leaders are the ones who’ll drive change. I invited the young Masters, PhDs and postdocs from my community (the Computational Biodiversity Science and Services training program – BIOS²) to design the change they wanted to see in peer review. Some of them have never performed a review before, and it was delightful to see them discovering that they don’t need to wait to be invited – they can contribute to science by commenting on preprints, on specific topics they know about, or specific sections.

From June to November 2023, we met once a month to discuss what it looks like to write kind peer reviews in Ecology and Evolution. In the first meeting, we “reverse-engineered” a good review by going through the PREreview’s Open Reviewers Review Assessment Rubric and brainstorming what we thought was missing from the rubric, thinking about the specificities of our research area. In the following meetings, we reviewed a total of five preprints, all published in PREreview, and had a chat with Daniela Saderi about watching our biases when reviewing scientific products, and with Kristen Thyng about reviewing research software.

By our final meeting, we all had a clear understanding of how preprints and preprints reviews fit into the academic publishing system and felt more empowered and encouraged to review scientific manuscripts. With this knowledge, we got together to brainstorm our own guide for kind peer review in Computational Ecology, which is now published and publicly available on Zenodo.

The Community Project support from ASAPbio was instrumental in keeping participants motivated throughout this journey of learning by practice, and in promoting a welcoming and warm environment where we all felt comfortable to question, comment, and fail forward. The small seeds will now flourish in our own communities, where we’re hosting preprint review clubs, kind peer review training, and talking about preprints. These young leaders are now paving the way for a kinder and more collaborative future in science. 

Application details

ASAPbio welcomes applications for support to convert existing journal clubs into preprint review clubs (local versions of crowd preprint review). Support is provided in the form of a gift card to cover refreshments for journal club attendees. It is expected that each club will produce 4 reviews of preprints that will be publicly posted to the platform of choice (e.g. PREreview or as a comment on the server). Applications are open to departmental journal clubs and individual lab journal clubs.

ASAPbio will provide converted journal clubs with a guide on how to run a crowd preprint review club and are available to deliver a session about preprints to the journal clubs. Converted journal clubs will be highlighted on the ASAPbio website on a map with links to the relevant departments. 

Eligibility & agreement

Applicants:

• Agree to abide by the ASAPbio Code of Conduct in conducting project activities. 
• Must be part of organising a journal club and have a verifiable profile on the internet
• Agree to update ASAPbio on the publishing of each review and to be contacted for a potential survey of the experience or contribute to a blog post at the end of the year
• Agree to upload at least 4 crowd reviews over the course of the year. Reviews may be uploaded to PREreview, added as a comment to the preprint article or shared in a different manner so long as the reviews are permanently publicly accessible.

Expenses:

• Funds will be provided by ASAPbio staff in the form of a gift card (of the applicants choosing) to the value of $200 
• Please note that the gift card must be appropriate for refreshments and that Amazon or similar services will not be accepted. Examples of appropriate locations include local supermarkets or deli’s

Applications will be processed on a rolling basis. If you have any questions please get in touch (jonny.coates@asapbio.org)

Frequently Asked Questions

1. Is the application open to journal clubs around the world? Yes, applications are accepted from any country
2. Who chooses the preprints to review? The journal club is free to choose any preprint they wish. There is no external invovlement in the selection of preprints, writing or posting of the reviews
3. Do the authors of posted reviews need to be identifiable or can reviews be shared anonymously? We do not dicate how you share your reviews. We encourage open and transparent reviews, which would include adding the names of authors to the posted reviews. However, this is not essential and if a journal club would prefer to post reviews anonymously then that is acceptable. The only requirement is that ASAPbio are able to receive notification of the reviews posted for tracking purposes.
4. Where can I post the preprint reviews? We do not dicate where you should post the reviews, so long as they are public. This could be as a comment on the preprint, using hypothesis.is or a review platform such as PREreview.
5. What is the format of the reviews? Again, the exact format is at the discretion of the journal club. It is anticipated that the journal clubs will be behave as a local form of our crowd preprint review; individual members may only comment on small sections of a preprint but together, the journal club produces a comprehensive review, akin to a journal-organised peer review. ASAPbio can provide a template for the final review if required.

Post by ASAPbio Fellow Aditi Sengupta

Preprints are increasingly becoming a tool to support the peer-review process and aid rapid dissemination of research results. The increased transparency in the review process that preprints can support has been welcomed by many journals with many pivoting to an environment of supporting preprints. A few examples include editorial policies that allow or encourage deposition of manuscripts at preprint servers (e.g. Springer Nature journals), partnering with platforms like Review Commons which provide journals with high-quality referee reports to reduce re-review rounds and shorten the publishing process (e.g. Review Commons affiliate journals), and partnering with preprint servers to allow authors to submit preprints to both the journal submission system and the preprint server (e.g. Development, PLOS). 

Another innovative way to engage with preprints is for journals to facilitate community review of the submitted manuscripts on their own website. For example, all European Geosciences Union’s (EGU) journals post manuscripts as preprints on the journals’ discussion forums for public peer review in the form of invited referee comments, author comments, editor comments, and community comments. This allows journal-coordinated reviews and open community reviews to take place in parallel, with authors responding to the reviews publicly, contributing to an open discussion, and supporting a transparent peer-review process. This opens up the peer-review process to the broader scientific community, and potentially integrates community reviews and editor-coordinated reviews to deliver a polished peer-reviewed article. 

To understand the operations of this peer-review model, we interviewed a former, and a current, associate editor of Biogeosciences as part of the 2022 ASAPbio Fellows program. Biogeosciences is an open-access EGU journal that publishes papers on the interactions between the biological, chemical, and physical processes in terrestrial or extraterrestrial life. As a current associate editor, Dr. Denise Akob handles submissions in astrobiology, exobiology, biogeochemistry, and geomicrobiology. Dr. Akob noted that EGU’s discussion-based process has many positives, including “lifting the veil” of the peer-review process. Public discussion/posting of review and responses allows all stakeholders to objectively understand the process, and is especially valuable to early career researchers for whom the process may have training and educational value. Dr. Akob noted that open peer review “forces reviewers to be constructive and kind”, and to constructively structure feedback to improve the science. The public discussion piece is particularly helpful in creating an atmosphere of conversation between authors, editors, reviewers, and the community, thereby opening a space for dialogue. 

An open peer review process also “potentially encourages authors to submit higher quality work since the feedback will be public”, notes Dr. Ben Bond-Lamberty, a previous associate editor at the journal. There is a possibility that such open reviews, when not positive, can affect the credibility of the study and/or the authors but Dr. Bond-Lamberty highlights how “having more eyes on the study has the likelihood of improving the research, and improves the odds of detecting data manipulation or other problems”. 

Dr. Akob and Dr. Bond-Lamberty noted that in their editorial role, they do not request that invited reviewers take into account public comments, but as editors they do pay attention to these comments and may request the authors to respond to them. This may provide particular buy-in value to journals interested in innovating their peer-review process, given that community reviews available on the discussion board hosted by the journal itself can be more easily integrated into the official review process compared to reviews hosted on a third-party platform. 

When discussing preprints as a communication format for research findings, neither of the editors had encountered hesitancy towards preprints from the broader discipline-specific scientific community, irrespective of career stages. A challenge however arises with governmental agencies that may have additional approval steps in place to publish a preprint, this may be a deterrent to preprint for those who are employed in such agencies. 

The concept of preprints is fast gaining traction across disciplines and will continue to evolve as journals get a better idea of what works and what does not in the context of their editorial processes. For example, eLife recently announced (Eisen et al.) a new editorial process that eliminates acceptance/rejection decisions after peer review. Instead, eLife will publish every paper that it reviews as a Reviewed Preprint that includes the manuscript and the journal’s peer reviews, with the goal of producing public peer reviews that highlight the science and include an assessment of the rigor and significance of the findings. This model steps away from the traditional route of peer review and provides an indication that journals are ready to experiment and drive forward a transparent peer-review process that provides more autonomy to authors. 

Dr. Bond-Lamberty comments that

“the science publishing world will be a mix of preprints and traditional articles for the foreseeable future, but having open and quantifiable records of the review process will be increasingly common”

with the possibility of performing meta-research on the reviews. Analysis of such metadata including demographics of people engaging with the peer-review process and their career stages may yield relevant ethnographic details of publishing in the STEM fields. Some disciplines may lead and others lag in implementing models that incorporate preprints but an overall fine-tuning of the process stands to vastly improve the quality of papers, establish transparency, and support timely completion of the peer-review process.  

By Thomas Lemberger

This post originally appeared on reviewcommons.org.

Preprints with public peer reviews (a.k.a. “preprint+” or “refereed preprints”) provide a new publication route for scientists to rapidly communicate and access research that has been reviewed by experts. By posting a refereed preprint, researchers can demonstrate their peer reviewed research output at an early stage, before the final endorsement and publication by a journal. In addition, the peer reviews provide readers with a detailed and transparent assessment of the validity of the study and the strength of its conclusions. To accelerate the formal publication process, refereed preprints produced by Review Commons can be seamlessly transferred to 17 journals¹ published by EMBO Press, eLife, ASCB, The Company of Biologists, Rockefeller University Press and PLOS without starting the entire process anew, thus avoiding redundant rounds of peer review.

At Review Commons, posting reviews publicly is always accompanied by the possibility for authors to simultaneously post a detailed response to the points raised by the reviewers. While the peer review process aims at involving independent experts (“peers”) in providing a critical analysis (“review”) of a work to verify its validity, correct mistakes and improve it, maintaining a balanced scientific discourse is essential for a productive outcome. Review Commons therefore considers the right of authors to reply to critiques raised by reviewers as an integral and fundamental part of the peer review process.

In a recent survey, Review Commons’ authors were asked the question: “What is the earliest stage that would be acceptable to post the reviews?”. The results (1,095 respondents) show that many authors feel it is acceptable to post the reviews as soon as the refereed preprint can be transferred to a journal (see figure, below). The data also reveals that the ability to formulate a response to the reviewers is a critical consideration. These results guided the new Review Commons policy that leaves sufficient time for authors to prepare a thoughtful response, which is then posted together with the review when the refereed preprint is transferred to an affiliate journal for consideration.

Refereed preprints make new peer reviewed research quickly accessible to readers. But they should also provide tangible benefits to the authors in terms of academic credit and recognition. To this end, EMBO is launching a pilot with its EMBO Postdoctoral Fellowship Programme by extending the eligibility criteria to formally include refereed preprints (announcement: Refereed preprints recognized as eligibility criterion for EMBO Postdoctoral Fellowships). Specifically, a first author preprint with public in-depth peer reviews obtained from trusted independent peer review platforms will be sufficient for eligibility to the programme. More details on the EMBO Postdoctoral Fellowship Programme’s eligibility criteria and its preprints policy are available at https://www.embo.org/funding/fellowships-grants-and-career-support/postdoctoral-fellowships/

A legitimate concern about publicly posting reviews is that some reviews could include remarks formulated in a too dismissive or aggressive tone. Review Commons editors will screen reviews for inappropriate content before sending them to the authors and before public posting and request amendments from the reviewers when necessary. In addition, if authors strongly feel that serious issues remain with the reviews they receive, a formal procedure allows them to challenge a report in exceptional cases before making it public and transferring the manuscript to an affiliate journal (see more details at https://reviewcommons.org/authors). In cases of disagreement, it will also remain possible for authors to withdraw their manuscript from Review Commons without having to post the reviews.

It is still early days for preprint peer review, and we will carefully monitor the response to these important policy changes. Adjustments might be needed but it is already clear that we have entered an exciting phase where a new research object – the refereed preprint – opens new avenues in scientific communication and the practice of transparent peer review.

1. Affiliate journals: EMBO Journal, EMBO Reports, EMBO Molecular Medicine, Molecular System Biology, Life Science Alliance, eLife, Journal of Cell Biology, Molecular Biology of the Cell, PLOS genetics, PLOS biology, PLOS computational biology, PLOS pathogens, PLOS one, Journal of Cell Science, Development, Disease Models & Mechanisms, Biology Open.