| Story Views | |
| Now: | |
| Last Hour: | |
| Last 24 Hours: | |
| Total: | |
Where peer-review went wrong
[Note: this post is by Mike. Matt hasn't seen it, may not agree with it, and would probably have advised me not to post it if I'd asked him.]
The magic is going out of my love-affair with peer-review. When we started seeing each other, back in 2004, I was completely on board with the idea that peer-review improves the quality of the scientific record in two ways: by keeping bad science from getting published, and by improving the good science that does get published. Eight years on, I am not convinced that either of those things is as true as I initially thought, and I’m increasingly aware of the monumental amount of time and effort it soaks up. Do the advantages outweigh the disadvantages? Five years ago I would have unhesitatingly said yes. A year ago, I’d have been unsure. Now, I am pretty much convinced that peer-review — at least as we practice it — is an expensive hangover from a former age, and does more harm than good.
What’s that? Evidence, you say? There’s plenty. We all remember Arsenic Life: pre-publication peer-review didn’t protect us from that. We all know of papers in our own fields that should never have been published — for a small sample, google “failure of peer-review” in the Dinosaur Mailing List archives. Peer-review doesn’t protect us from these and they have to be sorted out after publication by criticism, rebuttal and refinement. In other words, by the usual processes of science.
So pre-publication peer-review is not getting the job done as a filter. What about its role in improving papers that do get published? This does happen, for sure; but speaking as a veteran of 30 submissions, my experience has been that no more than half of my reviews have had anything constructive to suggest at all, and most of those that have improved my manuscripts have done so only in pretty trivial ways. If I add up all the time I’ve spent handling and responding to reviewer comments and balance it up against the improvement in the papers, my honest judgement is that it’s not been worth it. The improvement in the published papers is certainly not worth as much as all the extra science I could have been making instead of jumping through hoops.
And that of course is ignoring the long delays that peer-review imposes even in the best-case scenario of resubmit-with-revisions, and the much longer ones that result when reviews result in your having to reformat a manuscript and start the whole wretched process again at another journal.
All of this is much to my surprise, having been a staunch advocate of peer-review until relatively recently. But here’s where I’ve landed up, despite myself: I think the best analogy for our current system of pre-publication peer-review is that it’s a hazing ritual. It doesn’t exist because of any intrinsic value it has, and it certainly isn’t there for the benefit of the recipient. It’s basically a way to draw a line between In and Out. Something for the inductee to endure as a way of proving he’s made of the Right Stuff.
So: the principle value of peer-review is that it provides an opportunity for authors to demonstrate that they are prepared to undergo peer-review.
It’s a way of separating the men from the boys. (And, yes, the women from the girls, though I can’t help thinking there’s something very stereotypically male about our confrontational and oppositional review system.)
Finally, I should address one more thing: is this just whining from an outsider who can’t get in and thinks it’s a conspiracy? I don’t think so. I’ve run the peer-review gauntlet quite a few times now — my publications are into double figures, which doesn’t make me a seasoned professional but does show that I am pretty serious. In other words, I am inside the system that I’m criticising.
–
For full disclosure, I should make it clear that I am writing this a day after having had a paper rejected by reviewers. So if you like, you can write it off as a bitter ramblings of a resentful man. But the truth is, while this was the immediate trigger for writing this post, the feeling has been building up for a while.
Next time: some of the details of why my paper was rejected, and why I think they’re dumb reasons. In part three: what peer-review should actually be for, and what I plan to do with the paper now.
2012-08-05 18:00:35
Source: http://svpow.com/2012/08/05/where-peer-review-went-wrong/
Source:
