It’s peer review week and to celebrate, we are releasing a preview of one of the 2024 ASAPbio Fellows projects focused on investigating the ASAPbio Crowd Preprint Review activities.

ASAPbio Crowd Preprint Review

The theme of peer review week this year is “Innovation and Technology in Peer Review”. Since 2021, ASAPbio has been running its own version of crowd preprint review. This approach involves using a crowd of reviewers to collaboratively review a manuscript and was developed by the journal Synlett in 2017. The use of a crowd is the innovative aspect to this approach of performing peer review as it enables real-time discussion amongst reviewers and allows individual reviewers to only comment on aspects that fall within their expertise. This means that a large number of small comments can be combined into a traditional-looking peer review. Additionally, ASAPbio posts all reviews publicly which are subsequently linked back to the preprint, providing transparency and greater context for readers.

The ASAPbio approach involves crowd leads choosing preprints and disseminating them to the crowd members for review. Once crowd members add their comments, crowd leads synthesise the comments and discussions into a traditional-looking peer review. They then upload the synthesised reviews so that the reviews are publicly available (Fig 1).

ASAPbio began with a pilot in 2021 focused on cell biology preprints. The following year this expanded to three disciplines, including one in collaboration with SciELO that covered Portuguese-language preprints. In 2023, there were 4 crowds covering preprints across bioRxiv. Four crowds were again launched in 2024 and will run for an entire 12 month period, in contrast to the more tightly defined periods of previous years. Between 2021 and September 2024, this initiative has produced 97 public reviews on preprints (Table 1).

Experience of crowd members

A group of Fellows has been conducting a survey of current and past crowd members to reflect on the experience of being a crowd member. As a preview of our work, we present some data on the survey respondents and the reasons that they joined the program.

The majority of respondents indicated that they had a background in biology or meta-research with remaining disciplines being collated into “other”. Given the historical roots of ASAPbio as a life science organisation, it is unsurprising that biology is so well represented. There has also been a focus on meta-research as a crowd which explains the number of respondents who identified this as being their scientific area of interest (Fig 2A). In terms of career stages, most respondents were post-doctoral researchers (post-docs) or Principle Investigators (PIs) (Fig 2B). There were also 7 students who responded to the survey. Of those who stated that they had never participated in a crowd review, the majority were students or PIs whereas those most actively engaged were the post-docs. Those who identified as “other” job titles were also actively participating.

Why do people participate (or not) in crowd review?

Respondents indicated varied reasons for why they participate in crowd review activities (Fig 2C). Engaging in Open Science practices was most cited, followed by developing peer review skills, and contributing to the science community. Less frequently, respondents mentioned participating in crowd reviews for personal gains, such as increasing their visibility, showing evidence of their productivity, and keeping up-to-date with the latest research in their field. The top reasons for participating collectively represent 3 levels of impact: 1) Society, 2) Self, and 3) The scientific community. This strong emphasis on contributing to open science and self-directed training highlights the two key benefits of crowd preprint review. Indeed, the collaborative nature of the review process was highlighted in several free-text responses, with one responder communicating how it enables “sharing reviewers concerns, coming to an agreement, [and] sharing the workload.”

The two biggest reasons that respondents stated for not reviewing a given preprint were a lack of relevant expertise and a lack of time (Fig 2D). Given the size of the crowds, it is not unexpected that some will not have the specific expertise for any component of a given preprint. However, the benefit of a crowd is that this “expertise” burden is reduced as reviewers do not need to comment on the entire preprint. It is possible that this aspect could be better communicated to encourage people to review more granularly and to accept that as a valid contribution. Academics are under heavy workloads and are time-poor. Additionally, we have noticed in previous iterations of the crowd review program that in the busier months in the academic calendar, there is an anecdotal reduction in reviewing activities. A lack of interest in the topic was the third most common reason for not reviewing a given preprint. Thus far, crowds have been designed to cover entire fields (such as “cell biology” or “immunology”), so it cannot be expected that every chosen preprint will be of interest to the entire crowd.

This preview of our on-going efforts highlights altruism and a desire to participate in better practices as the primary motivator for participating in the crowd review program.

Conclusion

We’re still performing a full data analysis and expect to post a full preprint before the end of the year! But if you would like to join the crowd preprint review efforts for 2024-2025 you can sign up by clicking this link.

I was tired of hearing students and colleagues sharing their bad experiences with reviewers. I was tired of working really hard to submit a paper for publication and receiving inconsiderate reviews and editorial decisions. We needed a change of culture in peer review: change it from being just a “Where’s Waldo” game of pointing out flaws and what the paper didn’t do towards fair criticisms based on what’s presented, and also reinforcing the strengths of the work. I believe that peer review can be kind and fair, while still being rigorous.

As with any cultural change, we need to focus on the new generation, the leaders to come, rather than the leaders in place. The new leaders are the ones who’ll drive change. I invited the young Masters, PhDs and postdocs from my community (the Computational Biodiversity Science and Services training program – BIOS²) to design the change they wanted to see in peer review. Some of them have never performed a review before, and it was delightful to see them discovering that they don’t need to wait to be invited – they can contribute to science by commenting on preprints, on specific topics they know about, or specific sections.

From June to November 2023, we met once a month to discuss what it looks like to write kind peer reviews in Ecology and Evolution. In the first meeting, we “reverse-engineered” a good review by going through the PREreview’s Open Reviewers Review Assessment Rubric and brainstorming what we thought was missing from the rubric, thinking about the specificities of our research area. In the following meetings, we reviewed a total of five preprints, all published in PREreview, and had a chat with Daniela Saderi about watching our biases when reviewing scientific products, and with Kristen Thyng about reviewing research software.

By our final meeting, we all had a clear understanding of how preprints and preprints reviews fit into the academic publishing system and felt more empowered and encouraged to review scientific manuscripts. With this knowledge, we got together to brainstorm our own guide for kind peer review in Computational Ecology, which is now published and publicly available on Zenodo.

The Community Project support from ASAPbio was instrumental in keeping participants motivated throughout this journey of learning by practice, and in promoting a welcoming and warm environment where we all felt comfortable to question, comment, and fail forward. The small seeds will now flourish in our own communities, where we’re hosting preprint review clubs, kind peer review training, and talking about preprints. These young leaders are now paving the way for a kinder and more collaborative future in science. 

Application details

ASAPbio welcomes applications for support to convert existing journal clubs into preprint review clubs (local versions of crowd preprint review). Support is provided in the form of a gift card to cover refreshments for journal club attendees. It is expected that each club will produce 4 reviews of preprints that will be publicly posted to the platform of choice (e.g. PREreview or as a comment on the server). Applications are open to departmental journal clubs and individual lab journal clubs.

ASAPbio will provide converted journal clubs with a guide on how to run a crowd preprint review club and are available to deliver a session about preprints to the journal clubs. Converted journal clubs will be highlighted on the ASAPbio website on a map with links to the relevant departments. 

Eligibility & agreement

Applicants:

• Agree to abide by the ASAPbio Code of Conduct in conducting project activities. 
• Must be part of organising a journal club and have a verifiable profile on the internet
• Agree to update ASAPbio on the publishing of each review and to be contacted for a potential survey of the experience or contribute to a blog post at the end of the year
• Agree to upload at least 4 crowd reviews over the course of the year. Reviews may be uploaded to PREreview, added as a comment to the preprint article or shared in a different manner so long as the reviews are permanently publicly accessible.

Expenses:

• Funds will be provided by ASAPbio staff in the form of a gift card (of the applicants choosing) to the value of $200 
• Please note that the gift card must be appropriate for refreshments and that Amazon or similar services will not be accepted. Examples of appropriate locations include local supermarkets or deli’s

Applications will be processed on a rolling basis. If you have any questions please get in touch (jonny.coates@asapbio.org)

Frequently Asked Questions

1. Is the application open to journal clubs around the world? Yes, applications are accepted from any country
2. Who chooses the preprints to review? The journal club is free to choose any preprint they wish. There is no external invovlement in the selection of preprints, writing or posting of the reviews
3. Do the authors of posted reviews need to be identifiable or can reviews be shared anonymously? We do not dicate how you share your reviews. We encourage open and transparent reviews, which would include adding the names of authors to the posted reviews. However, this is not essential and if a journal club would prefer to post reviews anonymously then that is acceptable. The only requirement is that ASAPbio are able to receive notification of the reviews posted for tracking purposes.
4. Where can I post the preprint reviews? We do not dicate where you should post the reviews, so long as they are public. This could be as a comment on the preprint, using hypothesis.is or a review platform such as PREreview.
5. What is the format of the reviews? Again, the exact format is at the discretion of the journal club. It is anticipated that the journal clubs will be behave as a local form of our crowd preprint review; individual members may only comment on small sections of a preprint but together, the journal club produces a comprehensive review, akin to a journal-organised peer review. ASAPbio can provide a template for the final review if required.

Awareness of preprints in Africa

The adoption of preprints across the globe has not been equitable, with most preprints being posted by the USA, UK, Western Europe and China. Regions such as Africa have a low level of preprinting. Unfortunately, data around the usage, awareness and perceptions of preprints in Africa is lacking. Therefore a group of Fellows undertook a survey to investigate the awareness and perceptions of preprints amongst Africa based researchers. The survey received over 180 responses and will culminate in a preprint posted to AfricaArXiv. 

Preprints and graduation requirements

Across the world graduation requirements for PhD programs vary from no requirements to having published in multiple journals. Additionally, within the same institutions, programs can be just as variable. A further compounding issue is that requirements for graduation are not always formal requirements, often existing as expectations from departments or supervisors. Moreover, where requirements exist, they are often not transparent. 

To begin to tackle this issue, a group of 2023 Fellows undertook a global survey to gather information on the graduation requirements of master-level and PhD programs. This group gathered over 80 responses across 18 countries and produced a detailed blog post including recommendations on graduation requirements. 

Awareness of preprints in India

Similarly to Africa, the adoption of preprints across India has lagged behind the USA, UK, Western Europe and China. In 2022 ASAPbio ran a workshop on research assessment and preprints in India. Building from this, in 2023 the ASAPbio Fellows, in collaboration with India BioScience held 3 workshops to gather a deeper understanding of the attitude and perspectives of preprints in India. The outcome of these workshops is partially communicated in a blog post (link to follow) and will inform some of our activities in 2024. 

Preprints in progress

One of the major benefits of preprints over the traditional publishing model is that preprints allow authors to share work at earlier stages, including as iterative updates. To date, there are no studies assessing the number or use of iterative preprints, although there are a small number of known examples. A group of 2023 Fellows produced a preprint investigating potential iterative preprints. A second group of Fellows produced an infographic outlining the alternative uses of preprints, beyond the traditional journal article styled output.  

Crowd preprint review

In 2021, ASAPbio started activities to facilitate public reviews on preprints inspired by the crowd review model pioneered by the journal Synlett. In the first year, 14 public reviews of cell biology preprints were posted. In 2022, crowd review activities expanded to include cell biology, biochemistry and infectious diseases preprints (in Portuguese from SciELO Preprints). This resulted in 27 public reviews for bioRxiv preprints and 13 reviews for SciELO Preprints. In 2023, there were 4 crowds led by ASAPbio Fellows covering neuroscience, cancer biology, metabolism and meta-research; together, these crowds posted 35 public reviews of preprints. These reviews are available from sciety and PREreview:

• ASAPbio Cancer Biology Crowd
• ASAPbio Metabolism Crowd
• ASAPbio Meta-Research Crowd
• ASAPbio Neurobiology Crowd

Crowd preprint review will return next year in an expanded format. Additionally, in 2024 ASAPbio will provide support for current departmental journal clubs to convert into preprint review clubs in a further effort to expand preprint peer review and help train the next generation of scientists. 

2023 Fellows

I want to thank all of the 2023 ASAPbio Fellows for their contributions over the year. They have provided valuable insights into preprints across different geographical locations and communities, produced preliminary data on graduation requirements and further expanded the crowd review. They have been a source of inspiration to bring positive change in scholarly communication. I look forward to continuing to interact with the Fellows across other ASAPbio initiatives and within the community.

Looking to 2024

We’re undertaking plans to revitalize the Fellows program for 2024 and look forward to opening applications soon. If you’d like to be the first to know when the applications open, sign up to our newsletter and follow us on social media. If you have any questions please email jonny.coates@asapbio.org. 

The agenda

The event will kick off with introductions and a presentation from the preprint author. This will be followed by an interactive discussion about the preprint. Discussion points will be synthesized and posted as a comment on the preprint.

The preprints

• Monday, Dec 4, 12-1pm | Melissa Hoyer (Harvard Medical School) | Combinatorial selective ER-phagy remodels the ER during neurogenesis
• Tuesday, Dec 5, 12-1pm | Tara Essock-Burns and Dave Matus (Arcadia Science) | Gotta catch ’em all: Agar microchambers for high-throughput single-cell live imaging
• Tuesday, Dec 5, 1-2pm | Marcus Harrell (Boston College) | The Arp2/3 complex promotes periodic removal of Pak1-mediated negative feedback to facilitate anticorrelated Cdc42 oscillations
• Tuesday, Dec 5, 2-3pm | Piyush Nanda (Harvard University) |  Multicellular growth as a dynamic network of cells

Directions will be circulated to confirmed participants. Lunch will be provided, and attendance is capped at 8 people to ensure a highly interactive discussion.

We hope to see you there! To RSVP, fill in the form.

During our September Community Call, we heard from two publishers, eLife and PLOS Biology, about their experiences with preprint peer review models. 

The first speaker, Fiona Hutton, Head of Publishing at eLife, discussed eLife’s new publishing model. Fiona started by introducing problems with the current publishing system. Apart from being slow, the publishing system is undervaluing academics’ time and doesn’t give authors control over their work by requiring them to perform the work requested by the reviewer, which can often take months to complete or require multiple revisions, thus delaying the publication. She also pointed out that the current system gives higher value to where the article was published than the content of the article itself.

Fiona contrasted this with eLife’s new model. In this model, the submission of a manuscript is followed by the editor’s decision whether to send the article for review. Following review, reviewers, and editors consult with each other and reach a consensus, which is published as eLife’s assessment. The eLife assessment comments on the significance of the findings and the strength of the evidence. Before the reviews are published, authors are given a chance to address them. At this point, authors can revise the manuscript based on reviewers’ comments (most authors follow this route) or leave it unchanged. Whatever the decision is, eLife publishes the manuscript with the assessment as a Reviewed Preprint. 

Fiona emphasized the benefits of the eLife model. The process is much faster, and it takes 1-3 months. The average time from submission to publication of the Reviewed Preprint is 79 days compared to 170 days in the previous eLife model. The process is transparent, with detailed assessments, comments, and all manuscript versions publicly available. Authors also have more control over the process. They can publicly respond to reviewers’ comments and decide whether they want to revise the manuscript. 

When the model was presented to the public, it gained much attention, both criticism and praise. After several months since its implementation, submissions are steadily increasing, showing that there is an interest and authors see value in Reviwed Preprints. 

Our second speaker was Nonia Pariente, Editor in Chief at PLOS Biology.

Nonia started by reminding the audience about the objective of PLOS to increase the adoption of open science practices among the PLOS authors and beyond. One of the Open Science practices is the adoption of preprints. 

Nonia emphasized that PLOS was always supportive of receiving submissions that were preprinted and one of the early adopters of the bioRxiv to Journal and Journal to bioRxiv workflow, and is a supporter of meta-research on preprints.

PLOS’s commitment to preprints is also supported by its collaboration with preprint peer review platforms such as Review Commons and Peer Community In. Review Commons is a journal-independent preprint peer-review platform. When authors decide to submit to Review Commons, their preprints are peer-reviewed by reviewers selected by Review Commons. The preprint and reviews are then posted on bioRxiv. Authors can decide to send their manuscript with the reviews and their response with a revised version or a revision plan to any journal that is part of the Reviews Commons network, which includes PLOS journals. 

Nonia presented data from this collaboration. PLOS received 526 submissions through Review Commons, and have published over 200 papers, with PLOS Biology being the most popular among PLOS journals. She also noted that 70% of submissions (90% for PLOS Biology) don’t require additional reviewers, and papers received through Review Commons have shortened publication time by around 30%.

PLOS also works with Peer Community In. The collaboration between PLOS and PCI is similar to the collaboration with Review Commons. However, so far, PLOS has received fewer manuscripts through PCI. When a preprint is sent to Peer Community In, it is reviewed and recommended by the PCI’s specific topic community. Authors can then send it to a PCI-friendly journal such as PLOS. At the end of the presentation, Nonia emphasized her support for Publish Your Reviews, an ASAPbio initiative encouraging reviewers to publish their reviews publicly alongside the preprint. She believes there is a lot of effort wasted during peer review, and if somebody wrote a peer review, they own it and can publish it. 

The talks were followed by a discussion about preprints, peer review, and the future of scientific publishing. 

We would like to thank our speakers and all the event participants. If you would like to learn more, but couldn’t attend the event, follow the link below to watch the recording.