Re: [asa] Nature: Peer Review Trial and Debate

On 6/15/06, George Murphy <gmurphy@raex.com> wrote:
>
> There are certainly debatable aspects of the peer review system for
> scientific publications. They are being discussed by scientists in those
> publications themselves.
> E.g., there has been ongoing discussion in Physics Today (which is not
> itself a peer reviewed journal but is an official publication of the
> American Institute of Physics which published a number of such journals).
>
> My comment here is simple. Anyone who has not been involved in the process,
> both as an author of such papers and as a referee for them, probably doesn't
> know enough about it to be able to comment on it intelligently.
>
> Shalom
> George

The point I was trying to make to Janice was to show her what peer
review looks like by citing an open reviewed paper. I have found
double blind peer review -- my experience is in the engineering realm
so there may be some differences with basic science -- works
reasonably well. What double blind review looks like is the reviewer
and reviewee do not know each other's identity. The rationale behind
this is to make sure the reviewer provides more candid feedback -- the
very political effect that Janice is concerned about. The downside is
the reviewer can be peevish and trivial in their reviews because their
name is never attached.

The reason why open review has become a hot topic is some famous fraud
cases. Normal peer review is not a good guard against this and I don't
see open review doing any better. In time, fraud works itself out as
results do not get replicated. This latter part is underappreciated by
most lay people. They see something in the scientific press and only
see when the first reports come out before they are replicated. In the
scientific community it is only when the results are replicated that
it has credence. For example, J Hendrik Schoen had fradulent papers in
nanotechnology which needed to be withdrawn in part because they
couldn't be replicated at IBM's TJ Watson lab. Randy, were you
involved with this when you were with IBM?

My own thinking is that we have a semi-open review. That is the names
are not revealed until the paper is published or rejected and both the
review and paper get published simultaneously with names attached.
This would also allow for a better handling of embargos then having
the document prematurely published prior to the full review. We
already have the science press exagerating and misrepresenting the
results of papers when they are released in publication. This would
only get worse if they troll the public web sites prior to the review
finishing.

All this being said, openness is usually much better than secrecy and
an excellent motivator against bad behavior. As we get more examples
of open review documents I expect Janice to put up or shut up showing
how peer review is just a political game rather than a useful, but
limited, tool to improve the quality of scientific research. Peer
review is to science what Winston Churchill said about capitalism.
It's not perfect but there really is no better alternative.
To unsubscribe, send a message to majordomo@calvin.edu with
"unsubscribe asa" (no quotes) as the body of the message.
Received on Thu Jun 15 13:23:09 2006