Peer-Review Flaws, circa 2002
People familiar with what happened in climate science during the last year might find Lawrence K. Altman’s NYT article “THE DOCTOR’S WORLD; When Peer Review Produces Unsound Science” of June 11, 2002 more than prescient (emphasis all mine, of course):
[…] Yet for all its acclaim, the system [of peer-review] has long been controversial. Despite its system of checks and balances, a number of errors, plagiarism and even outright fraud have slipped through it.
[…] A particular concern is that because editors and reviewers examine only what authors summarize, not raw data, the system can provide false reassurances that what is published is scientifically sound.
[…] Researchers reported [in the “The Journal of the American Medical Association” in June 2002] considerable evidence that many statistical and methodological errors were common in published papers and that authors often failed to discuss the limitations of their findings. Even the press releases that journals issue to steer journalists to report peer reviewed papers often exaggerate the perceived importance of findings and fail to highlight important caveats and conflicts of interest.
[…] Because the anonymous peers chosen to review manuscripts are often the authors’ scientific competitors, jealousies and competitive advantage can become factors in the reviews.
[…] The peer review system also tends to set a very high barrier for authors to publish truly novel findings.
[…] Yet research on peer review has found that many studies are conducted without the benefit of adequate consultation with statisticians, sometimes because none were available.
[…] Once statistical errors are published, it is hard to stop them from spreading and being cited uncritically by others. […]