{"id":3645,"date":"2018-01-24T00:00:00","date_gmt":"2018-01-24T00:00:00","guid":{"rendered":"http:\/\/pl-asapbio.local\/eisen-appraise\/"},"modified":"2025-03-28T21:38:13","modified_gmt":"2025-03-28T21:38:13","slug":"eisen-appraise","status":"publish","type":"post","link":"https:\/\/asapbio.org\/eisen-appraise\/","title":{"rendered":"APPRAISE (A Post-Publication Review and Assessment In Science Experiment)"},"content":{"rendered":"

By Michael B. Eisen<\/em><\/p>\n

Investigator, Howard Hughes Medical Institute<\/em>
\n Professor of Molecular and Cell Biology, UC Berkeley<\/em>
\n mbeisen@gmail.com<\/a>, <\/em>@mbeisen<\/em><\/a><\/p>\n

With the rapid growth of bioRxiv, biomedical research is entering a new era in which papers describing our ideas, experiments, data and discoveries are made available to our colleagues and the public without having undergone peer review. This inversion of the traditional temporal relationship between peer review and publication is good for science and anyone who cares about or benefits from its accomplishments. But it also has rendered the existing infrastructure we have for carrying out peer review \u2013 scientific journals \u2013 obsolete.<\/p>\n

It is incumbent upon to scientific community to seize this opportunity, reinventing the ways we assess our research outputs and each other to make them more fair, efficient and effective. This will require new ideas and a lot of experimentation. In that spirit I describe here a new project \u2013 called Appraise<\/i> \u2013 that is both a model and experimental platform for what peer review can and should look like in a world without journals.<\/span><\/p>\n

Background<\/b><\/p>\n

Scientific journals once served a useful purpose. In the age of ink and paper, it made sense to bundle manuscripts based on audience and to screen submissions before incurring the cost of printing dissemination via the postal service. Remarkably, while the logic for organizing science communication this way evaporated with the birth of the Internet, scientific journals have survived their obsolescence, their anachronistic role in distribution and collation sustained by the stranglehold they have on the process of peer review.<\/p>\n

The mainstreaming of preprints in biomedicine has brought the distribution of scientific knowledge into the 21st century. Preprints give scientists control over when to publish their work, eliminated needless delays and barriers to access that impede scientific process and the realization of its benefits.<\/p>\n

But we are currently in a weird situation where the scientists increasingly use preprints to communicate their ideas and discoveries, but still rely on Gutenberg era journals as the sole vehicle for assessing the validity, import and audience of works of science, and judging the contributions of the researchers whose ideas and effort they represent.<\/p>\n

This is bad for science. Journals are expensive \u2013 we spend around $10 billion per year<\/a> to keep them in business. They are slow \u2013 it takes an average of nine months to publish a paper. And most importantly, the system of peer review they oversee does not work very well.<\/p>\n

When everything can be published with the click of a button, it is nonsensical to structure peer review around deciding what works get into which publications.  It makes no sense to evaluate the impact of a paper only at the time of its initial publication. It makes no sense to limit the assessments that matter to the two or three people a journal was able to coax into serving as reviewers. It makes no sense to reduce something as complex and multidimensional as the contributions and impact of a paper and its likely audience to a binary decision to publish in a particular journal. And it is bad for science to rely on a system whose entire premise is that  the most important thing about a paper is where it was published, and not what it actually says.<\/p>\n

Why does a field based on empiricism and rationality use a system that is so manifestly ineffective and illogical? It is not that people disagree with this assessment. One of the few constants in my nearly thirty years as a scientist is that everyone complains about nearly every aspect of peer review. Yet we seem trapped by the Churchillian view that our current system of peer review is the worst possible system, except for all of the others.<\/p>\n

I do not believe this is true. The rise of preprints gives us the perfect opportunity to create a new system that takes full advantage of the Internet to more rapidly, effectively and fairly engage the scientific community in assessing the validity, audience and impact of published works.<\/p>\n

Axioms of Post Publication Peer Review<\/b><\/p>\n

I do not want to underestimate the challenges in creating and operating a robust system of peer review that would work robustly for the ~2 million scientific papers published every year. Problems with the journal based, pre-publication peer review are manifest, but the system does have one virtue: it exists and functions.<\/p>\n

The devil is ultimately in the details of course. But there are some general principles that should govern the development of a new system for scientific publishing.<\/p>\n

    \n
  1. Researchers decide when their work is published. <\/b>When scientists have new ideas, data or conclusions they think will be valuable to their colleagues they should be able to share them without the mediation of a journal. Posting a preprint is not a prelude to publication; it is publication. There should be a basic screen for appropriateness \u2013 not every document with scientific content is a work of science \u2013 but once a work passes this screen it should be considered published. This act of publishing should generate an identifier and should be permanent and irreversible. Publishing new versions of works should be encouraged.<\/li>\n
  2. Peer review after publication<\/b>. An obvious corollary of author driven publishing is that all peer review will happen after publication. Many of the issues with journal driven peer review stem from it occurring exclusively prior to publication. Post-publication peer review, if done right, can be far more effective. This will require that people no longer equate a work being published with it having gone through peer review.<\/li>\n
  3. Peer review is a non-exclusive annotation. <\/b>One problems with journal based peer review is that it is exclusive the the journal in which it was published, and thereby excludes any and all other inputs. If instead we focus on reviewing works that have already been published, there is no limit on who can review a paper. All reviews are new annotations that add to the body of curation developed around every paper.<\/li>\n
  4. Peer review is ongoing<\/b>. Another absurdity of journal based review is that the assessment of a paper \u2013 recorded in the title of the journal in which it was published \u2013 never changes, even as people read the paper, use its methods and data, and view it in the ever changing context that is science. It should be possible to review papers at any point during the useful lifetime of a work, and for new reviews to contribute to an evolving understanding of the validity, audience, value and impact of the work.<\/li>\n
  5. Peer review is open.<\/b> Everyone should be able to participate in the process of peer review. There need to be systems to guard against conflicts of interest and various forms of gaming. The default should be for reviewers to be identified, but there also need to be mechanisms to allow reviews where the identity of the reviewer is not made public.<\/li>\n<\/ol>\n

    APPRAISE<\/i><\/b> (A<\/b> P<\/b>ost-P<\/b>ublication R<\/b>eview and A<\/b>ssessment I<\/b>n S<\/b>cience E<\/b>xperiment)<\/p>\n

    It is easy to rant about the evils of journals and philosophize about a better system in theory (I have been doing it for twenty years). But it has proven to be incredibly difficult to successfully launch alternative peer review systems in the presence of an existing system that dominates hiring, funding and promotion in academic science.<\/p>\n

    However the increasing number of unreviewed preprints, and the impact they are beginning to have on science communications, has created a unique opportunity to experiment with new systems. Guided by previous experiences with new initiatives (especially PLOS) to avoid trying to do everything all at once, I am launching an experiment in the post-publication review of papers published in bioRxiv called Appraise<\/i>.<\/p>\n

    The founding mission of Appraise<\/i> is to demonstrate the potential for post publication peer review to make the assessment of research publications and scientists more efficient, fair and effective. We will do this by:<\/p>\n