This morning I saw a new blog post by Peter Coles dealing with a topic I have written about before: the problems with the current system of scientific peer review. He too has strong opinions on the subject, which he has expressed several times on his blog. For the most part, I think our shared disgruntlement with the system stems from the same source — academic publishers trumpeting Peer Review (capitals intentional) as some sort of absolute certificate of quality that only they are able to provide, and provide at an exorbitant cost at that, whereas in fact not only it can probably be organised far more cheaply, the current system isn't all that hot at sorting out the wheat from the chaff anyway.
Coles takes a particular example of a paper recently published in the Astrophysical Journal in order to illustrate his point. The authors make a strong claim — the sort of strong claim that requires watertight evidence to back it up — and yet a key figure in the paper relies on interpreting a curve, which is supposed to be a cubic spline interpolation of the data, that passes through only two data points. Where are the other data points? Presumably they exist, but they aren't displayed in the figure. At the very least this is sloppy and unhelpful to anyone trying to reproduce the results, and the referee should have asked for a modification. But referees are often sloppy and distracted themselves; when there is only one referee to pass judgement on the merits of a paper the outcome of Peer Review should really be taken with a giant pinch of salt.
Read the rest at Blank On The Map
Coles takes a particular example of a paper recently published in the Astrophysical Journal in order to illustrate his point. The authors make a strong claim — the sort of strong claim that requires watertight evidence to back it up — and yet a key figure in the paper relies on interpreting a curve, which is supposed to be a cubic spline interpolation of the data, that passes through only two data points. Where are the other data points? Presumably they exist, but they aren't displayed in the figure. At the very least this is sloppy and unhelpful to anyone trying to reproduce the results, and the referee should have asked for a modification. But referees are often sloppy and distracted themselves; when there is only one referee to pass judgement on the merits of a paper the outcome of Peer Review should really be taken with a giant pinch of salt.
Read the rest at Blank On The Map
