New Models of Peer-Review: Implications for the Different Roles Journals Play in Scholarly Communities

Presenter: David Solomon, Michigan State University
Time: 2:55 PM - Thursday, July 12th 2007
Location: SFU Harbour Center Sauder Industries Policy Room

Traditionally, peer review is conducted by a group of “experts” selected by the editors, and both authors and reviewers are often blinded as to the identity of those on the other end. The new practice of open peer-review brings transparency to the review process. For example, Biomed Central (BMC) publishes the entire review process.

Using this format, the public can see and make their own decisions about the published research. This also allows anybody who is interested in sharing their research results to publish an article.

Reference was made to a piece written by Ann C. Schafner (1994), The future of scientific journals: Lessons from the past, Information Technology and Libraries 13:239-247. This article discusses the roles of journals in scholarly communities as:
1. Building a collective knowledge base
2. Communicating information
3. Validating the quality of research
4. Distributing rewards (evaluate the researcher)
5. Building scientific communities (discussions in editorials, commentaries)

The peer review process in relation to journal roles was explored, based on the above principles:
1. Building a collective knowledge base (essential)
2. Communicating information (detrimental)
Note: It typically takes 18-months for the conversation to happen
3. Validating the quality of research (essential)
4. Distributing rewards (essential)
5. Buildling scientific communities (irrelevant)

Exploring examples of open peer-review processes:
A. Nature
Between the period of June to September 2006, the highly regarded basic science journal, NATURE, conducted a trial of an open peer-review process as an experiment. Nature typically receives 10,000 paper submissions which undergo a 2-step process of initial screening then selected papers are subject to peer-review. During this trial period, papers that made it through the initial screening was published in an open peer-review process

B. PloS (Public Library of Science)
PloS One has been established using an open peer-review policy. Basically, submitted articles are screened by an editor who looks at the paper for major problems, then publishes it immediately. Readers can annotate articles, as a form of social commenting. PloS One has been operating for about 7 months, and as of the 3rd of week of June 2007, 1,326 manuscripts have been published. It was also noted that the author fee for publication is approximately $1200.

C. arXiv.org
arXiv.org is a site representing the subject area of physics. On arXiv.org there are up to 5000 submissions per month. Taking a random sample, it was found that 444,079 connections were made on a particular day. This site has now expanded to math and quantitative biology.


Results
For Nature, the experiment was unsuccessful, despite initial enthusiastic responses. For 1,369 eligible papers, 71 (5%) of authors agreed to participate in the new open peer-review process. There were a total of 92 comments. 33 of the papers (46%) did not receive comments at all. In terms of readership, the site had received an average of 5,600 page views/week, but readers were simply not leaving comments. Of the comments received, a large proportion were not particularly helpful.
In contrast, PloS One and arXiv.org have been very successful with operating on an open peer-review system.

Analysis
Why do arXiv.org and PloS One work, but the Nature experiment did not?
Perhaps the reason is because there is something specific to Nature, as a journal itself. It was suggested that many authors, particularly in biology, were afraid of being “scooped” by participating in open commentary. Another explanation may be the fact that a publication in a highly-regarded journal like Nature bears such a significant influence on authors’ careers, that they choose not to take a chance and participate in a new experimental review process.

Examples of other specialty journals that have implemented open peer-review:
Electronic Transactions on Artificial Intelligence (http:www.ida.liu.se/ext/etai/)

Atmospheric Chemistry and Physics (http:www.copernicus.org/EGU/acp/)

The MEO (Medical Education Online) pilot project (http://www.med-ed-online.org/index.html)

Medical Education Online is an open access journal that has been operating for 11 years, providing physicians and other health professionals with medical research results. MEO is currently in the process of implementing a hybrid open review system. Briefly, initial screening of submissions is conducted after which manuscripts are published in a special “preprint” section of the journal. Readers are able to post comments and authors have the option of replying to these comments. It was noted that all comments are first screened by an editor before being posted. Concurrently, a regular peer-review process is conducted. After approximately 6 weeks, the review editor decides on the acceptance of a manuscript for publication based upon the feedback from both the public and selected peer-reviewers. Accepted manuscripts are then published in a separate “peer-reviewed” section of the journal. Preprints and comments to non-accepted manuscripts are kept online unless requested to be removed by the author.

Preliminary results from MEO pilot
20 of 27 (74%) agreed to participate in the open peer-review
Average between 1 and 2 comments per article but the quality of the comments tend to be reasonably good, in many cases on par w/ solicited reviews
The biggest value is getting the information out there quickly and out to the public

Comments from the audience:
Many have experienced “reviewer fatigue”. There are rewards for authors in the number of published articles, but reviewers are seldom compensated or rewarded for their efforts. Could a system be established in which merit can be earned through providing reviews?

In the system of open commenting, a person needs to feel adequate to provide insightful comments. Perhaps there are very few who feel qualified to comment in Nature, unless officially asked to do so?

Also, the social phenomenon of responsibility diffusion may be occuring in open online commenting. People tend to think someone else will provide comments, but when a reviewer is formally selected, they then have the responsibility to provide constructive feedback.

Related commentary link: www.webcitation.org/5OC7DQBzQ

In Summary
Acceptance of open peer-review by authors vary greatly by journal and subject area. Also, the quality of the feedback does not necessarily reflect effectiveness of the model. Linda J. Miller, Nature’s U.S. executive editor as quoted in the Chronical of Higher Ed. Thursday, January 11, 2007: “ The next generation may be more comfortable with this process”. By utilizing open peer-review to increase access and participation in the review process, a form of crowd-sourcing is developed to advance intellectual scholarship. In addition, the invitation of the public into the discussion creates a much more inviting atmosphere which heightens the incentive for society to be innovative, something that is stifled in the traditional peer-review process.

Perhaps it is as Max Planck once suggested, “opposition dies and the new generation accepts”.