20.459 peer-reviewing

               Humanist Discussion Group, Vol. 20, No. 459.
       Centre for Computing in the Humanities, King's College London
  www.kcl.ac.uk/schools/humanities/cch/research/publications/humanist.html
                        www.princeton.edu/humanist/
                     Submit to: humanist_at_princeton.edu

   [1] From: "Hunsucker, R.L." <R.L.Hunsucker_at_uva.nl> (144)
         Subject: RE: 20.457 peer-reviewing?

   [2] From: Michael Dunn <Michael.Dunn_at_mpi.nl> (21)
         Subject: Re: 20.457 peer-reviewing?

   [3] From: "Jochen L. Leidner"
<jochen.leidner_at_ed.ac.uk> (11)
         Subject: Re: 20.457 peer-reviewing?

   [4] From: "Alex Murzaku" <lissus_at_gmail.com> (11)
         Subject: Re: 20.457 peer-reviewing?

--[1]------------------------------------------------------------------
         Date: Wed, 21 Feb 2007 06:48:25 +0000
         From: "Hunsucker, R.L." <R.L.Hunsucker_at_uva.nl>
         Subject: RE: 20.457 peer-reviewing?

> Comments?
>
> Yours,
> WM

OK.

That's a nice rhetorical flourish of Armstrong's at the end (: not
"whether", but "how") but one which may throw a too facile
mantle of simplicity over a pretty complex matter. [One of his
academic specialties *is* "persuasive advertising", I see :-).]
Involving, for example, questions of transparency and legitimization
and negotiation of that "how", vis-à-vis the author but also the
reader and the knowledge domain. There are also more mundane
factors having to do, for instance, with stability, access,
bibliographical control, preservation (I add, as one on the library
side of things).

Fresh from an experience "as one reviewed" in Willard's words, I
have at the moment less than positive feelings about the matter. My
contribution was positively and even enthusiastically judged by the
anonymous reviewer and subsequently accepted. However, that
reviewer's secondary comments suggested in my view a peculiar lack
of sensitivity not only to the deeper import of my paper but even
to certain fundamental problematics of the discipline itself. I'll not
venture a guess whether both of Willard's two categries ("human
failings and perversities") might be involved here.

But apart from all of the above : if we really want to problematize
the phenomenon in philosophical and ideological ways, I'd suggest
laying the emphasis not only on the "review" part but certainly also
on that "peer" part. What do we mean in fact by that concept in this
day and age ? And if we get the concept clear in the postpositivist,
transdisciplinary, holistic, genre-blurring context of today, how do
we then operationalize it ??

And finally, Willard brings in again that natural sciences / humanities
contrast (with which I in other respects have big problems, as previously
said), in terms of competition and speed. But I'd invite you to consider
also other factors that may enter into this picture. One is the varying
emphasis put on matters of choice (and rigor) in methodology and for
statistical processing approach (here I think first of the biomedical sector),
and on the elimination of "bias" (formally defined). Related are the
differing visions regarding the factor subjectivity. Not unimportant as
well is the question of vested interests (economic, political, social). All of
these factors can be very differently at play in the "exact" sciences (more)
and the humanities (mostly much less) -- not least in the real-world-
functioning of the "peer review" system, in the dissatisfaction with it,
and in the criticisms leveled at it.

All this endless ado about the virtues and vices of peer review tends to
beg the actual question what one in fact wants to accomplish with such
an instrument. But there are various legitimate while quite different answers
to that question, differing in the disciplinary dimension but also in other
dimensions -- and therefore no univocal solution available to the identified
problems. We can be pretty sure that propositions like the nonsensical
"Peer Reviewing is one of the means for knowledge verification" or the
somewhat less idiotic "Peer Reviewing is one of the means for knowledge
validation" (see the KCPR 2007 website) will get us nowhere at all. [As
though all the efforts of all the epistemologists -- not to mention the poets
-- of the last several generations have been for naught.] Knowledge has no
need at all for this mechanism. What's at stake is not knowledge but in the
first instance prestige (legitimate prestige, one likes to suppose), and all
those good things that follow from that, and of which we are -- I feel
confident in saying -- all in varying degrees the beneficiaries, directly or
indirectly. All we really need is a perfect or at least a less deficient (and
perhaps less pretentious) alternate device for safeguarding those "academic"
amenities we so value.

Just an off-the-cuff reaction.

- Laval Hunsucker

P.s.: And if I may be extremely nitpicking : It's "Wharton" with an "h"
in there, which is also pronounced, no differently than in the word just
following the first comma in this sentence.

--[2]------------------------------------------------------------------
         Date: Wed, 21 Feb 2007 06:48:55 +0000
         From: Michael Dunn <Michael.Dunn_at_mpi.nl>
         Subject: Re: 20.457 peer-reviewing?

Dear Willard,

The 2nd International Symposium on Knowledge Communication and Peer
Reviewing is one of the "spam conferences" organised by the infamous
Nagib Callaos. His conferences are known for accepting computer
generated gobbledygook (papers are published at a fee per paper accepted,
no attendance required). There was a lot of discussion of it last year
-- here are a couple of good summaries:

http://3dpancakes.typepad.com/ernie/2005/04/sci_followup.html
http://itre.cis.upenn.edu/~myl/languagelog/archives/002067.html

I can't see anything more to the kcpr2007 website than an attempt to
defend the his business model from deserved mockery. Which reminds me of
another contribution to the literature on peer review:

http://www.theonion.com/news/index.php?issue=4116&n=3

All the best, Michael Dunn

-- 
Michael Dunn
Center for Language Studies, Radboud University, and
Language and Cognition Group, MPI Psycholinguistics
PB 310, NL-6500 AH Nijmegen, The Netherlands
   +31 (0)24 3521268 (wk) +31 (0)24 3521300 (fax)
   http://www.mpi.nl/Members/MichaelDunn
--[3]------------------------------------------------------------------
         Date: Wed, 21 Feb 2007 06:50:53 +0000
         From: "Jochen L. Leidner" <jochen.leidner_at_ed.ac.uk>
         Subject: Re: 20.457 peer-reviewing?
Hi,
There's a good and entertaining article in the 
"Squibs & Discussions" section of a recent CL issue [1] on this topic.
Surely double-blind peer review is limited, but 
the same applies in other realms of life: use the 
mediocre until you've got something better [2]. 
Democracy and capitalism have obvious flaws, 
however in the absence of better alternatives, we tend to stick with them. ;-)
         Regards,
                 Jochen
[1] Church, Ken (2005) 'Reviewing the Reviewers' Computational
     Linguistics 31(4): 575-578.
     Online: <http://portal.acm.org/citation.cfm?id=1110825.1110831>.
[2] Kassirer, Jerome P. and Edward W. Campion (1994) 'Peer Review: Crude
     and Understudied, but Indispensable' JAMA 272: 96-97.
     Online: <http://www.ama-assn.org/public/peer/7_13_94/pv3089x.htm>
>Even so, I wonder if it is fair to put the blame on peer-reviewing,
>which is bound to turn up human failings and perversities. I also
>wonder if, given our greater diversity in ways to publish now that
>the Web provides them, a cogent solution to whatever problems is to
>use that diversity. J. Scott Armstrong (Warton School, Penn) argues,
>for example,
>
> >Peer review improves quality, but its use to screen papers has met
> >with limited success. Current procedures to assure quality and
> >fairness seem to discourage scientific advancement, especially
> >important innovations, because findings that conflict with current
> >beliefs are often judged to have defects.
-- 
Jochen L Leidner
Royal Society of Edinburgh Enterprise Fellow
Language Technology Group, School of Informatics, University of Edinburgh
http://www.iccs.inf.ed.ac.uk/~s0239229/
--[4]------------------------------------------------------------------
         Date: Wed, 21 Feb 2007 06:50:53 +0000
         From: "Alex Murzaku" <lissus_at_gmail.com>
         Subject: Re: 20.457 peer-reviewing?
I read a wonderful article from Ken Church on this issue a few years
ago. I do remember that it had a lot of arguments, but I can't
remember them and I can't locate my copy of Computational
Linguistics. I found its reference though and I hope it can be useful
to those that have access to CL's online archives:
Church, K. 2005. Reviewing the Reviewers. Comput. Linguist. 31, 4
(Dec. 2005), 575-578. DOI=
http://dx.doi.org/10.1162/089120105775299131
Regards,
Alex Murzaku
[See the reference elsewhere in this issue of Humanist --WM]