Monday, November 03, 2008

Peer review survey

Here is a link to an online survey being conducted by EPSRC, soliciting feedback on their peer review process, used for research grant proposals. Dissatisfied as I am with this process, I felt the urge to contribute. (The survey's closing date is November 7th.)

The general problem with their procedure for peer review, is that it is very primitive, in comparison with the process by which conference submissions are reviewed. You send in a grant application, you get 3 or 4 reviews, you get to submit a response, and these get considered by the panel. A specific problem with this is the shortage of interaction amongst the parties. Suppose that two reviewers think the proposal is excellent, and another thinks it's rubbish. The correct thing to do, surely, is to get the reviewers to settle their differences via a discussion, and preferably solicit another review. This is exactly what program committees do when papers are selected for conferences. In some conferences that I have been involved with, the authors of a submitted paper make a response to the reviews, the reviewers then continue their discussion in the light of the author's response. The whole thing is much more interactive, and helps to ensure transparency and accountability.

Given that it is more significant to get a research grant than to publish a paper (both for the grant applicant/author, and for the taxpayer) it is bizarre in the extreme that EPSRC still uses such a primitive system. There is a case for still more effort being directed towards ensuring that good choices are made. For example, perhaps research grant applicants should be interviewed by the panel (however, suggestions like that go beyond the scope of the above mentioned survey).


Kai Arzheimer said...

Misgivings about the validity of the peer-review process are by no means confined to the sciences. Together with a number of colleagues I have recently launched a web survey on the practice of peer-reviewing manuscripts for political science journals. Our (extremely preliminary) results point to widespread worries about conflicts of interests.

Paul Goldberg said...

Thanks for that comment. Generally I think we have to trust in peer review to do its job, and trust people to decline to review when they have a conflict of interest. But, for EPSRC proposals specifically, the process of moderating and reconciling differences of opinion, leaves much room for improvement.