Peer Review: ‘Herding’ Behavior Versus ‘Gut’ Instincts in Science

An interesting article in NATURE scrutinizes the peer review process via a computer model (1). The authors compare the rapidity of acceptance of a hypothesis in a field when peer review is based solely on objective analysis of data versus when the reviewer includes subjective feelings about the validity of the hypothesis. They find that convergence is more rapid in the first case. Rapid convergence has an element of herd behavior- this may be valuable when the hypothesis under consideration is indeed correct. However, if it is not, then ‘herding’ can lead to premature acceptance of false concepts.

The NATURE article reflects a problem that is very familiar to scientists undertaking review of journal articles (or grant proposals). In the current ideal model of peer review one is supposed to confine oneself to the data and methods presented in the submitted article. If these all seem consistent then one should recommend acceptance of the article by the journal.  However, often one has the ‘gut’ feeling, based on long experience, that something isn’t right, that the results can’t be valid given the nature of the experiment or the methodology used. But you can’t just come out and state this in your review! In these cases reviewers often hunt for some excuse to reject the article, but are sometimes forced into reluctant acceptance. If more subjectivity were allowed in peer review it might reduce the frequency of papers, especially high profile ones, whose data cannot be reproduced upon subsequent analysis (2). This type of subjective evaluation does take place- in hallway conversations at meetings or in the university cafeteria, but perhaps there should be more of a place for it in the formal review process. Eventually flawed research is revealed- especially if the topic is quite important. However, reputations are not made by repeating the work of others, and surely here are many observations and concepts of more modest importance that persist in the literature even though they are basically incorrect.   

(1) http://www.nature.com/news/peer-reviewers-urged-to-speak-their-minds-1.14302

(2) http://blogs.nature.com/news/2011/09/reliability_of_new_drug_target.html