cscheid, 10 Jun 2015

(Equal parts reflection, armchair-quarterbacking, and fantasy-writing about why peer review feels ineffective, on both the giving and receiving end. More academic navel-gazing, you’ve been warned.)

Maybe too much has been said about why peer review is broken, but today I want to highlight a point I don’t think I see discussed as often, which is that by the time a reviewer gets to a paper, it might be too late.

What I mean by that is that even though most papers have a kernel of a good idea, at some point before submission the process took a turn for the worse that makes the submission unsalvageable. What’s a reviewer to do? I’ve been on both the giving and the receiving end of this.

In the giving end, reading the paper feels like this: “I see.” “Yes, good point” “Ok, so now you’re going to… what? huh? no. No. What? wait, don’t bother with this experiment, what no no ughhhh”. After three separate sections of this, the reviewer tends to write a review amounting to “the writing is unclear and the motivation is missing. I don’t understand the experiments and also they didn’t cite random reference X”. This is obviously a bad review, because it all points to local fixes that can be done, rather than global fixes that should be done. In the reviewer’s mind, those are not the reason the paper is bad, but they’re a reason for the paper not to be accepted, and it’s easier to write the latter than the former. I can definitely relate to the feeling of despair of reading a paper and going “oh dear, if only I had gotten to this six months ago”.

In the receiving end, reading the review, this is how feel: “This review says nothing substantial about the paper, and the reviewer clearly did not like it. It seems like even if all of these fixes were done, the reviewer still would have rejected the paper. What now?”.

Like the ER doctor treating a hyperglicemic crisis wishes they could fix the problem by controlling the patient’s sugar intake in the last 10 years but is limited to insulin shots in the present, the reviewer gets a paper that needed 12 months of care, and writes “I guess you can fix the typo on Theorem 4.2” or “experiments do not compare against technique Y”. Eventually the patient is discharged from the ER, but the long-term damage is sort of done.

Preventive health care for papers

This mess is partly the fault of the reviewing systems that ask for “concrete ways in which the paper can be improved”. But I think the real problem is deeper. The reviewer can do the most good at the beginning of the process, and at the end of the process, the best reviewers can do is pick out the few submissions that happened to avoid all the pitfalls. But if we know what the pitfalls are, why not address them at the best time? This is where journals can actually add a lot of value to a submission.

Authors announce, early in the process, their intention of doing work on an area, and submit this to a journal. The journal editor then assigns a reviewer early in the process. The reviewer gets to nudge the paper’s progress in this way or that, and hopefully the total gnashing of teeth is significantly reduced. This can even be done in the open if you take the Github-overlay-as-arxiv-of-the-future perspective: have reviewers file bug reports early on. Journals now serve as the clearing house for this process; good journals will earn a reputation for good fostering; good editors find good matches between authors and reviewers, and everyone wins. Forget gatekeeping or filtering, and think of journals as academic public health. We shouldn’t be trying simply to find the best work: we should be trying to make as much good work as possible.

Of course this is tricky because early in the process is the best time to steal an idea; but this kind of stealing already happens anyway late in the process, so the cat’s kind of out of that bag. And if you think having a reviewer early on increases the power of the reviewer too much, that just tells me you haven’t been in either side of a knock-down drag-out reviewing war (that we’re waging on each other forever more).

So besides path dependence, why do we not do this? Maybe this larger degree of involvement is what the Ph.D. process should be about (or should have been about in the bad cases). But then the fostering process would provide a secondary opportunity for best practices to diffuse through a community.