A column in last week’s SCIENCE described a study by Michael Lauer of the NIH who compared NIH peer review of the merits of grant proposals to the publication record and impact factors of the investigators who submitted those proposals. He found essentially no correlation between high evaluation by the NIH study section and future publication productivity. This result has apparently triggered some hand wringing at the NIH and calls to alter the way in which study sections work. However, this concern may be misplaced.
Every scientist who is worth his/her salt can recount experiences where an NIH study section trashed a grant on a topic that later turned out to be quite important. It has been pointed out many times that NIH study sections are inherently conservative and tend to reflect the existing consensus in a field. Major breakthroughs, however, come from science that disrupts consensus. This has been understood ever since Thomas Kuhn and The Structure of Scientific Revolutions. The issue is how to identify breakthrough science and differentiate it from misguided error. To give the NIH some credit, in recent years it has implemented a number of granting programs that try to stress innovation (e.g. the Pioneer awards) as well as funding for beginning investigators who may have some new ideas.
Almost the converse situation prevails in publication where the ilk of Nature, CELL and other high profile journals try to focus exclusively on ‘hot’ topics. Over time some of the designated hot areas prove to be not so hot and dwindle away, but nonetheless the ‘hot’ articles will have generated high impact factors. No wonder there is a discrepancy between NIH funding and publication impact!
Although there is no need to obsess about the grant/publication disparity, it is clear that the NIH peer review system could use some renovation (as well as more money to dispense). The titles and functions of study sections still primarily reflect a disease/organ system specific orientation. However, current biomedical science is developing information and insights that cut across traditional boundaries. If the study section system were more reflective of the thrust of current research there would probably be fewer misfires on grant funding.