{"id":3623,"date":"2018-03-30T00:00:00","date_gmt":"2018-03-30T00:00:00","guid":{"rendered":"http:\/\/pl-asapbio.local\/amaral-quality\/"},"modified":"2025-03-28T21:38:06","modified_gmt":"2025-03-28T21:38:06","slug":"amaral-quality","status":"publish","type":"post","link":"https:\/\/asapbio.org\/amaral-quality\/","title":{"rendered":"Comparing quality of reporting between preprints and peer-reviewed articles \u2013 a crowdsourced initiative"},"content":{"rendered":"

By Olavo B. Amaral<\/strong>
\n Institute of Medical Biochemistry, Federal University of Rio de Janeiro, Brazil<\/strong><\/p>\n

As the preprint movement gains traction in biology, the time is ripe to revisit some aspects of scientific publication that we view as fundamental \u2013 first and foremost of which is the peer review process itself. Common concerns about preprints include the possibility that they may be less reliable than peer-reviewed articles<\/a>, and thus diminish the average quality of the literature. On the other side of the debate, preprint supporters have been trying to set up journal-independent platforms for peer review<\/a>, in order to incorporate elements of traditional peer review within the preprint ecosystem.<\/p>\n

All of this seems to indicate that preprints have not changed the belief of most scientists that peer review is somehow vital to the scientific enterprise. That said, do we actually know how much peer review actually adds to published papers, and how much difference there is between preprints and peer-reviewed articles?<\/span><\/p>\n

As a matter of fact, we don\u2019t know much. For a community that believes so fervently in peer review, and prides itself on being guided by empirical evidence, we have done surprisingly little to measure the benefits we extract from it. All in all, it wouldn\u2019t be unfair to say that those \u201cIn Peer Review We Trust\u201d signs at the March for Science are ultimately exchanging one dogma for another, with not that much evidence to support either of them.<\/p>\n

A 2007 Cochrane review<\/a> on editorial peer review concluded that \u201clittle empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research.\u201d Studies in specific journals<\/a> comparing pre- and post-review versions of articles have mostly shown peer review to improve them on average, but the benefits tend to be small. Automated comparisons<\/a> between preprints on arXiv and their peer-reviewed published versions based on text metrics also suggest that changes are typically not major.<\/p>\n

On the other hand, opinions of reviewers on articles and grant applications have been shown to have little<\/a> or no<\/a> agreement among themselves, suggesting that the process has a large degree of subjectivity. Experiments with intentionally flawed articles also suggest that both methodological errors<\/a> and completely unsound papers<\/a> easily slip through the peer review system, leading to the corollary that, due to the sheer amount of journals with varying peer review quality, nothing can currently deter poor-quality science from making it to publication<\/a>.<\/p>\n

Although it is surprising that so little research is available to measure the actual benefit provided by peer review, this might have something to do with the difficulty of measuring the quality of published science<\/a> \u2013 as attested by the very low agreement on reviewers\u2019 subjective assessments of papers and projects. Nevertheless, we believe that there are at least some objective and measurable dimensions of an article\u2019s quality, and that these are conceivably the ones that editorial peer review can improve the most.<\/p>\n

Reporting quality for methods and results is one of these objective dimensions, which unsurprisingly has been the focus of many attempts to improve the published literature. This is best exemplified by reporting guidelines for specific types of articles, such as the CONSORT statement<\/a> for clinical trials and the ARRIVE guidelines<\/a> for animal experiments, many of which are currently compiled by the EQUATOR Network<\/a>. Checklists have also been used by individual journals<\/a> upon article submission to try to improve reporting quality and increase reproducibility.<\/p>\n

Recently, we have seen at least one major effort to test the effectiveness of this kind of intervention. The NPQIP study<\/a> retrospectively compared papers in Nature Publishing Group\u2019s journals before and after such a checklist was implemented, concluding that the proportion of articles reporting on randomization, blinding, exclusions and sample size calculations increased \u2013 although most of them still lacked information on these topics. A similar study, IICARus<\/a>, is currently being run by the same group in order to evaluate the impact of the ARRIVE guidelines.<\/p>\n

If thoroughness in reporting methods and results can be used to evaluate the effect of reporting guidelines, why shouldn\u2019t we use it as a proxy to examine the contribution of the peer review process itself? Inadequate reporting, after all, should be one of the most straightforward problems to be picked up by peer review, and also one of the easiest to fix. Moreover, many aspects of reporting quality can be assessed by objective measures, thus decreasing subjectivity in evaluation.<\/p>\n

With this in mind \u2013 and very much inspired by the crowdsourcing approach used by the NPQIP and IICARUS studies, we have started a project to compare reporting quality between preprints in bioRxiv and published articles in PubMed. The comparison aims to answer two main questions: (1) whether the average preprint differs from the average peer reviewed article in terms of reporting quality and (2) whether the average reporting quality of articles changes between their preprint and published versions.<\/p>\n

For the first question, we are comparing random samples of bioRxiv and PubMed papers from 2016 \u2013 a process that is already under way (see figure). To be included, articles must be written in English and present original results with at least one statistical comparison among groups composed of humans, nonhuman animals, cells or microorganisms (or from biological samples derived from them). For the second question, we plan to compare the same bioRxiv papers with their own published versions, in order to assess the changes brought on by the peer review process.<\/p>\n

To reduce bias and engage a wider community in the study, we are using a crowdsourced approach, recruiting scientists who wish to engage in the project as article evaluators. We already have eight registered evaluators, but are looking to enroll more people to speed up the process. If you are a researcher in the life sciences and would like to participate, we will send you a test sample of 4 papers, a link to an online form and an instruction manual providing orientation on the various questions \u2013 although most of them are rather straightforward. After passing the test, participants will receive two papers a week to evaluate, with each taking around 30-45 minutes to complete in our hands.\"\"<\/p>\n

Besides helping to answer two vital unanswered questions in our current scientific ecosystem, all evaluators will be included as authors in the project and any literature deriving from it. Thus, if you are interested in participating, let us know! If you need further information, you can take a look at our preregistered protocol<\/a> at the Open Science Framework to know more about the project. We are looking to include 76 articles in each group, each of them scored by 3 independent evaluators in order to reduce bias \u2013 which is a lot of work on a collective level, but not as much on the individual one if more people choose to join in.<\/p>\n

And what do the results look up to now? As the figure shows, it looks like a close tie \u2013 although we can\u2019t tell you which side is winning, as data analysis is blinded to experimental groups. That said, there\u2019s a lot of variation within each group, so we will definitely wait to reach our final sample before making any claims. If you want to be on board with us, please let us know by writing to olavo@bioqmed.ufrj.br<\/a> or clarissa.franca@bioqmed.ufrj.br<\/a>, and get involved ASAP!<\/p>\n

\n

\"\"Olavo Amaral is an associate professor at the Federal University of Rio de Janeiro and an ambassador for ASAPbio. Besides studying preprints, he is interested in neuroscience and meta-research, and currently heads the Brazilian Reproducibility Initiative<\/a>, a multicenter effort to systematic replicate results from Brazilian biomedical science.<\/em><\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

By Olavo B. Amaral Institute of Medical Biochemistry, Federal University of Rio de Janeiro, Brazil As the preprint movement gains traction in biology, the time is ripe to revisit some […]<\/p>\n","protected":false},"author":2,"featured_media":2444,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[42,49,44],"tags":[],"class_list":["post-3623","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-guest-posts","category-peer-review","category-preprints"],"acf":[],"_links":{"self":[{"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/posts\/3623","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/comments?post=3623"}],"version-history":[{"count":1,"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/posts\/3623\/revisions"}],"predecessor-version":[{"id":3624,"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/posts\/3623\/revisions\/3624"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/media\/2444"}],"wp:attachment":[{"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/media?parent=3623"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/categories?post=3623"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/asapbio.org\/wp-json\/wp\/v2\/tags?post=3623"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}