First, in some fields the rational content is very low. A spoof is
relatively easy to get past the "keepers." Where the empirical standards
are low, it's easier. I recall a prize-winning poet named Spectrum, if I
remember correctly, hailed as perhaps superior to sliced bread. The
glorious thoughts were produced with a one from column A, one from column
B, etc., approach. Or consider Crews' /The Pooh Perplex/, where he
parodies several schools of criticism very effectively.
Second, there are always cheats. There are those that fake data, the ones
usually pursued. But there are also reviewers who blacklist a proposal so
they can steal the ideas and submit them as their own. How can one prove
that a rascal stole his idea, especially since one can seamlessly
backdate a file?
Third, there are sometimes gross errors. I recall a statement by Szent
Georgi (hope I spelled that right) who said that for years he submitted
proposals based on the last set of results from his lab. This worked well
until one proposal was denied as impossible. He was too far ahead of the
field at the time. There can be a penalty for being too good.
Fourth, with all its problems, peer review is the best technique
available for the evaluation of reports and proposals.
Dave
On Thu, 25 May 2006 22:29:50 +0100 "Hon Wai Lai" <honwai.lai@gmail.com>
writes:
Does anyone want to comment on this article from UK's Financial Times
(www.ft.com) on limitations of peer review?
.........................................................................
..............................
John Kay: Conflicting opinion drives science
By John Kay
Published: May 22 2006 19:37 | Last updated: May 22 2006 19:37
The Royal Society, Britain’s scientific establishment, has just released
a report on public communication of scientific findings. Journalists in
search of stories and scientists anxious for publicity and research
funding issue early, oversimplified or downright misleading accounts of
research. Unsubstantiated claims of a link between immunisation and
autism have caused distress to millions of British parents. Korea’s
progress in stem cell research seems to have been won at the expense of
truth and ethics.
The Society’s answers are self-restraint and peer review. Peer review is
the process by which professions review their own work. Articles
submitted to journals receive critical assessment from referees
experienced in the field. Peer review is a bulwark against cranks, crooks
and incompetents. But too much reliance on peer review carries its own
dangers. Every profession defines its own concept of excellence in
inward-looking ways.
Successful academics learn how to trigger the buttons that win the
approval of referees. The physicist, Alan Sokal, demonstrated this by the
submission of a spoof article to the cultural studies journal Social Text
in 1996. The content was nonsense, but the form and jargon corresponded
so closely to reviewers’ expectations that the contribution was accepted.
Professor Sokal’s purpose was to demonstrate that standards were lower
and more subjective in softer subjects than in more scientific ones and,
while he was right, the problem identified was more general. All
subjects, from architecture to physics, from literary criticism to
economics, develop what Thomas Kuhn called paradigms – assumptions common
to all practitioners and assumed to represent universal truth until a new
paradigm displaces the old.
A further step down a well defined road wins easier acceptance than a
deviation from the beaten track. Most academic research is therefore
boring, and more so as scholarship has become more professional,
eccentricity less tolerated and peer review multiplied through processes
of grant awards and research assessment. The latest idea in Britain is to
make these processes routine by shifting from the costly and fallible
exercise of subjective judgment to a cheaper and objective system of
quantitative metrics. This can only aggravate the problems.
Big advances come through the paradigm shifts and peer review makes this
difficult. The line between the crank and the genius is sometimes a fine
one and may only be apparent after time has elapsed. Many Nobel Prize
winners had difficulty securing early recognition. The world of today
favours the competent professional – as judged by the standards of other
competent professionals. In a sense this self-reference is right: the
people to decide whether astrology is good astrology are other
astrologers. But they are not the people to decide whether astrology
itself is any good. Judgment of the rigour and relevance of professional
standards and scholarly research can never be left to professionals and
scholars alone.
The Tacoma Narrows Bridge, an elegant suspension bridge in Washington
State, carried traffic for four months in 1940. In a high wind, the flat
deck acquired a beautiful wave pattern. The oscillations grew larger and
larger until the roadway finally disintegrated into Puget Sound.
The trade newspaper, Engineering News-Record, was forced to retract its
suggestion that the designer, Leon Moisseiff, might have been
responsible. The editors apologised for any inference drawn by “the
casual reader” that “the modern bridge engineer was remiss”.
But the perspective of “the casual reader”, though not a substitute for
peer review, is as essential as the contribution of the little boy who
pointed out that the emperor had no clothes. Any form of censorship,
including self-censorship and censorship by fellow professionals,
encourages complacency and discourages innovation. The history of modern
scholarship is that, more slowly than we would wish, truth and new
knowledge emerge only from a cacophony of conflicting opinions.
The writer’s new book, The Hare and the Tortoise, is published on June 6
Received on Thu May 25 19:58:58 2006
This archive was generated by hypermail 2.1.8 : Thu May 25 2006 - 19:58:58 EDT