Open Peer Commentary:
        A Supplement, Not a Substitute, for Peer Review

> From: Erik Sandewall <ejs@ida.liu.se>
> Date: Fri, 30 May 1997 00:00:18 +0200
>
> ETAI is a concrete attempt to realize many of the ideas which have been
> discussed in the present email exchange, including author-side payment
> of publication costs, free access for all would-be readers, a
> re-thinking of incentives and of reviewing structures

One can only applaud and welcome such projects, so I hate to have to
sound a note of caution: Author-side payment is the right model for the
steady state of the learned periodical corpus once the transition has
been made to the new medium. But to attempt to charge authors in the
early years -- when paper still prevails, and the new medium (and
journal) is still struggling to establish its bona fides with wary
authors worrying about quality control, permanence and academic credit
-- is to risk an early and needless demise: The transition must
initially be supported by outright subsidy. Author-side payment
comes later.

> First Publication Archives (similar to "Preprint Archives", but with a
> guarantee that the articles remain unchanged for an extended period of
> time)...
> ETAI pioneers the principle of *posteriori reviewing*: the reviewing
> and acceptance process takes place *after* the article has been
> published... The intention is that ETAI's quality control shall be
> considerably *more strict and reliable* than what is done in
> conventional journals.

Again, one can only wish such projects well, but there are two prominent
problems in this approach (which has been contemplated by many, and
implemented in several cases I know of: I hope their Editors will
join in this discussion and describe what their experience has been
with "open peer review").

I will describe the two problems in a tick, but let me just say that on
this particular question I may have more relevant experience to draw on
than anyone else alive today, having umpired both peer review and open
peer commentary for two decades now: Only the late Sol Tax -- Founder and
Editor of the journal Current Anthropology four decades ago, and the
originator of its unique "open peer commentary" service, which I
shamelessly copied two decades later, in launching Behavioral and Brain
Sciences -- would have been better qualified to speak here than I.

But, with me, he would have warned: OPEN PEER COMMENTARY IS A
SUPPLEMENT TO PEER REVIEW, NOT A SUBSTITUTE FOR IT.

That was the first problem. Peer review is imperfect; it can no doubt be
improved upon, but alternatives should first be tested; and in testing,
one is well-advised to manipulate one variable at a time: Here we
are dealing with a change in medium (paper to electronic), a change in
economic model (subscription to author-side payment) and a change in
quality control mechanism (peer review to open peer commentary).

But let us assume ETAI will be subsidised initially and that the
electronic medium will be welcomed: Why should it not be the empirical
test-bed for open peer commentary as a substitute for peer review?

It can be. But now let me describe the second problem: Anyone who has
edited or reviewed for a refereed journal knows that conscientious
refereeing requires a good deal of time and attention -- of which there
is not much to spare in the world of a busy researcher. We nevertheless
give it, for free, whenever we are asked and are able to oblige,
because of a kind of golden rule (plus a bit of superstition about
reciprocity between reviewing for the journal and the journal accepting
our own papers).

What we all get in exchange for the yearly hours we each devote to
refereeing is a certified, quality-controlled literature, which we can
read and build upon with reasonable confidence (modulo the fallibility
of human peer review). For not only is our refereeing time limited, so
is our reading time. And we want to be able to know (and choose) when we
are reviewing and when we are reading.

When we are chosen to review, it is by an Editor who has judged that
our expertise is pertinent. So we do not review at random. Editors
select the manuscripts that are suitable for review (some are not, and
then our time is not wasted), and of course we may accept or decline.

Nor do we read at random: We restrict the papers we read and cite to
those that have appeared in refereed journals (although we often read
and very cite recent work after it has been accepted but before it has
appeared, because of the slowness of paper publication).

If open peer review were substituted for the classical refereeing
system, the distinction between reviewing and reading would vanish.
So would the confidence that we had had in a refereed literature and
the possibility of restricting our reading and citation to such an
authenticated corpus.

And what would take its place? Unrefereed "first publications" along
with links to self-selected comments on unrefereed manuscripts by -- by
whom, one wonders? Who has the time or the motivation to wade boldly
through unrefereed manuscripts and "review" them for the rest of us?
And would those intrepid souls be the "peers" with the pertinent
expertise, or just the ones with time to spare for such rummaging? And
if they do have all that time to spare, are we really ready to be
guided by their judgment?

That said, I still hope ETAI flies, and that I will just have been a
stodgy pedant in having held out for classical peer review.

Harnad, S. (1996) Implementing Peer Review on the Net:
Scientific Quality Control in Scholarly Electronic Journals. In:
Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic
Frontier.  Cambridge MA: MIT Press. Pp. 103-118.
http://cogsci.soton.ac.uk/harnad/Papers/Harnad/harnad96.peer.review.html

Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A
difficult balance: Peer review in biomedical publication.)
Nature 322: 24 - 5.

Harnad, S. (1985) Rational disagreement in peer review. Science,
Technology and Human Values 10: 55 - 62.

Harnad, S. (1984) Commentaries, opinions and the growth of scientific
knowledge. American Psychologist 39: 1497 - 1498.

Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in
scientific quality control, New York: Cambridge University Press.

Harnad, S. (1979) Creative disagreement. The Sciences 19: 18 - 20.