Peer review – implications of ‘corruption’

Two separate events that occurred at the start of this month have made me reflect once again on the many myths surrounding the ‘hallowed’ peer review process on which so much academic credibility is seen to lie.

First, I received an e-mail from a friend for whom I had written a reference in connection with a grant application that they had submitted to one of the UK’s Research Councils.  They had received the disappointing news that despite two strong references, a third referee had been highly critical of the proposal, casting aspersions on their professional expertise and on the quality of the proposed research.  I was appalled by this.  The research proposal was one of the best I have recently read, and from what the Research Council said of the comments of the ‘third’ referee, they seemed to me to be completely inappropriate.  Either the referee was ignorant of the research field, or they had vested interests in ensuring that this research was not funded.

By coincidence, at about the same time, the BBC picked up on an open letter sent by a group of scientists last July that also criticised the traditional peer review process, but this time with respect to journal articles.  As the BBC Science Correspondent Pallab Ghosh commented, “Stem cell experts say they believe a small group of scientists is effectively vetoing high quality science from publication in journals. In some cases they say it might be done to deliberately stifle research that is in competition with their own”. The 14 scientists had written that “Stem cell biology is highly topical and is attracting great interest not only within the research community but also from politicians, patient groups and the general public. However, the standard of publications in the field is very variable. Papers that are scientifically flawed or comprise only modest technical increments often attract undue profile. At the same time publication of truly original findings may be delayed or rejected”.  To try to overcome this, they proposed that “when a paper is published, the reviews, response to reviews and associated editorial correspondence could be provided as Supplementary Information, while preserving anonymity of the referees”.

Peer review is one of the fundamental principles upon which the edifice of academic reputation – and financial reward – is based.  However, the system is inherently  flawed, and I find it somewhat surprising that it still retains such power.  Six issues warrant particular consideration:

• First, peer review is based on a belief that ‘science’ is in some way value free; that individual prejudice, political beliefs, or social agendas have no effect on academics’ judgements as to the quality of research.  Whilst many academics do indeed try to reach impartial judgements about the quality of work that they review, they undoubtedly bring biases to such judgements as a result of their own lives and research practices.  Moreover, editors of journals and Research Council panels exercise immense power through their choices of whom to ask to act as referees for papers or grant applications.  Science is not, and never has been, value free.
• Academic status is in part based upon the number of citations a paper receives.  Academics thus seek to publish in the most prestigious journals that have high citation indexes.  For a very long time, cartels of academics have therefore operated, deliberately citing each other’s works so as mutually to raise their profiles and status. Academics are only human, and it is scarcely surprising that they operate in this way.  There is nothing exceptional about this.  Some of us may not think it right, but it happens.
• One way that new ideas can begin to find voice is through the creation of new journals.  However, these take time to become established, and when status relies so much on having papers published in the most prestigious journals, it remains very difficult for new approaches and ideas to find widespread expression in this way; rarely do the most eminent academics deliberately choose to publish in new and ‘unimportant’ journals!
• Those who run the major journals and sit on grant-giving Research Council Boards have immense power, and most do their very best to be fair in the judgements that they reach.  However, by definition, the peer review system is designed more to endorse existing approaches to intellectual enquiry, rather than to encourage innovative research.
• None of this would matter particularly, and could merely be dismissed as irrelevant academic posturing, if there was not so much money involved.  Academic prestige and income depend fundamentally on success in publications and grant applications.  The UK’s Research Councils thus invest some £2.8 billion annually in support of research, and it is crucial that this is dispersed wisely.  It is therefore extremely sad – albeit typical – that in the case of my friend who had their grant application rejected, there was no right of appeal against the decision.  Panel chairs and editors must have the guts to stand up and recognise when they see flawed decisions being made by referees.  It is thus extremely encouraging to see that some Research Councils, notably EPSRC, are trying to create exciting new ways to support research that do not place excessive emphasis on traditional peer review processes.
• Finally, there is now a good case for exploring alternative ways of judging research ‘quality’.  ‘Publishing’ papers openly on freely available websites, and then assessing their quality by the number of ‘hits’ that they get would, for example, be a rather more democratic process than that through which a small number of ’eminent’ academics judge their peers.  Of course this would be as open to abuse as existing systems, but at least it would present an alternative viewpoint.

We must debunk the myth that there is something ‘pure’ or ‘objective’ about academic peer review.  It is a social process, just like any other social process.  It has strengths and weaknesses.  For long, it has served the academic community well.  However, as the 14 stem biologists who raised the lid of Pandora’s Box implied, it is a system that fails to encourage the most original research, and instead supports the system that gave rise to it.  After all, that is not so surprising, is it?