Re: [asa] Nature: Peer Review Trial and Debate

Rich Blinne wrote:

> The reason why open review has become a hot topic is some famous fraud
> cases. Normal peer review is not a good guard against this and I don't
> see open review doing any better. In time, fraud works itself out as
> results do not get replicated. This latter part is underappreciated by
> most lay people. They see something in the scientific press and only
> see when the first reports come out before they are replicated. In the
> scientific community it is only when the results are replicated that
> it has credence. For example, J Hendrik Schoen had fradulent papers in
> nanotechnology which needed to be withdrawn in part because they
> couldn't be replicated at IBM's TJ Watson lab. Randy, were you
> involved with this when you were with IBM?
>
> My own thinking is that we have a semi-open review. That is the names
> are not revealed until the paper is published or rejected and both the
> review and paper get published simultaneously with names attached.
> This would also allow for a better handling of embargos then having
> the document prematurely published prior to the full review. We
> already have the science press exagerating and misrepresenting the
> results of papers when they are released in publication. This would
> only get worse if they troll the public web sites prior to the review
> finishing.
>
> All this being said, openness is usually much better than secrecy and
> an excellent motivator against bad behavior. As we get more examples
> of open review documents I expect Janice to put up or shut up showing
> how peer review is just a political game rather than a useful, but
> limited, tool to improve the quality of scientific research. Peer
> review is to science what Winston Churchill said about capitalism.
> It's not perfect but there really is no better alternative.

Sorry for the delay in responding. I was out of town for over a week at the
Templeton Foundation meeting and was unexpectedly out of email contact. Now
I'm sifting through hundreds of posts. I'll comment on the ASA email list
notes later.

Yes, Shoen's fraudulent paper came out while I was at Watson. As you
pointed out, we had a team in my area that worked pretty hard at replicating
the results, to no avail. Suspicions were raised not just by the inability
to replicate the results but some of the claimed values of the I-V curve
didn't seem right based on what we knew about those molecules. Then someone
at a university noted that by magnifying the published data in Shoen's
paper, the noise fluctuations in supposedly different experiments were
identical. Only the scale had changed. That broke it open. Should that
have been caught in the peer review process? Perhaps. But when you review
a paper, do you analyze the data in that degree? And indeed, in the long
run, the scientific process builds upon a collection of independent work,
not a single paper that could be erroneous.

I agree with you that peer review is not perfect and never will be but it is
nevertheless the best alternative. But it can be abused and it must be
watched with vigilance. In that regard, ASA must ensure that it's peer
review system for PSCF is not only timely but fair. We currently have some
concerns about timeliness and responsiveness and some folks think we may be
too willing to publish the entire spectrum of perspectives. But again, it's
the best system we have. We do need more members to be willing to review
papers. Contact Roman Miller if you are interested and able.

Randy

To unsubscribe, send a message to majordomo@calvin.edu with
"unsubscribe asa" (no quotes) as the body of the message.
Received on Sun Jun 18 14:42:19 2006