Humanist Discussion Group, Vol. 13, No. 536.
Centre for Computing in the Humanities, King's College London
<http://www.princeton.edu/~mccarty/humanist/>
<http://www.kcl.ac.uk/humanities/cch/humanist/>
[1] From: Matt Kirschenbaum <mgk@pop.uky.edu> (14)
Subject: Re: 13.0531 evaluating WWW materials
[2] From: cbf@socrates.berkeley.edu (83)
Subject: Re: 13.0531 evaluating WWW materials
[3] From: "P. T. Rourke" <ptrourke@mediaone.net> (45)
Subject: Evaluating WWW Materials
--[1]------------------------------------------------------------------
Date: Sat, 08 Apr 2000 14:08:11 -0400
From: Matt Kirschenbaum <mgk@pop.uky.edu>
Subject: Re: 13.0531 evaluating WWW materials
On the original question, the MLA's Draft Guidelines on Evaluating Work with
Digital Media in the Modern Languages may be helpful:
http://www.mla.org/reports/ccet/draft_ccet_guidelines.htm
But I think the issue could also be considered not from the standpoint of
whether non-peer-reviewed materials should or should not count as "research,"
but rather what constitutes peer-review in the first place. In Eric Johnson's
example:
> As far as documentation of use and of acceptance by the scholarly
>community, the faculty member might list the sites that contain links to
>the material -- along with any annotation (for example, "The best
>web-based corpus of words of historical Spanish texts can be found at
>http:// ...").
"The best web-based corpus . . . " Is this not a form of peer-review? Is
peer-review necessarily limited to the mechanisms in place on the editorial
board of a journal or university press? Matt
--[2]------------------------------------------------------------------
Date: Sat, 8 Apr 2000 11:15:42 -0700 (PDT)
From: cbf@socrates.berkeley.edu
Subject: Re: 13.0531 evaluating WWW materials
UC Berkeley's Academic Senate adopted language last year that essentially
eliminates the difference in format among faculty research publications:
web, print, CD-ROM will all be considered equally. What did _not_ change
was the evaluation requirement, essentially peer-reviewed vs.
non-peer-reviewed.
I would argue that the web, in particular, allows us to get beyond that
dichotomy because it provides, potentially, the possibility of actually
measuring the impact of a particular web site on its discipline.
Unfortunately, the number of hits is as yet a fairly imprecise measure and
subject to manipulation ("Hey, I'm up for tenure; assign all your students
to look at my web site."). A better measure would be links _into_ a web
site. As Willard pointed out to me the other day, this is how the
Google.com portal works.
In some respects, this is like the traditional citation index that
measures the number of times a given study is cited in other studies.
What peer revieww does, essentially, is certify that three knowledgeable
members of a given discipline have read the piece of work and certified
that it's worthwhile. Considering the amazing amount of fairly useless
work that is published in peer-reviewed journal, such certification is a
fairly slender reed upon which to render academic judgments; but so far
it's the best we have.
[Of course, in the ideal case, peer review often improves the quality of
the piece of work reviewed.]
There is at least one initiative--the name escapes me at the moment--that
is attempting to divorce peer review from publication, sort of a Good
Housekeeping Seal of Approval that could be attached even to a
self-publication. It would be a Good Thing if the members of the
humanities computing community could devise such a system for ourselves
and our colleagues.
Charles Faulhaber The Bancroft Library UC Berkeley, CA 94720-6000
(510) 642-3782 FAX (510) 642-7589 cfaulhab@library.berkeley.edu
--[3]------------------------------------------------------------------
Date: Sat, 8 Apr 2000 20:46:52 -0400
From: "P. T. Rourke" <ptrourke@mediaone.net>
Subject: Evaluating WWW Materials
Sounds interesting as far as you go.
> We are in the process of revising the promotion and evaluation
> procedures in all of the departments at Illinois State University, and I
> have been asked by a committee to get input from individuals at other
> institutions concerning how non-peer-reviewed, web-based materials
> could/should be evaluated by the institution.
> Some of the best sources of online information on a variety of subjects
> have been created by faculty. There ought to be some means of giving the
> creators credit toward academic advancement.
Not from academic experience, but publishing experience, I'd suggest that
it's far too early to come up with a firm policy for web-based materials; it
would be better to just add some wiggle room. Ultimately a web-based
resource should be evaluated by "published" reviews rather than blind peer
review. The problem is that there aren't a lot of reviews available of
electronic resources. When they are available, one hopes that authors will
do their readers and evaluators the favor of linking to them.
So for now evaluation should be on a case-by-case basis. If the resource is
well-known and well used, the author should be able to get letters of
reference from users with some degree of expertise that can stand in for
published reviews until such time as those become available. A major
project should have the same cachet as a book or a series editorship; a
minor but well-appreciated project should have about the cachet of a
published paper of the same weight. And ultimately members of the same
department should have some sense of the scholarly quality of the work
without reference to other's reviews, unless you're talking about a
situation in which e.g. you have one Asian expert or Africanist in a History
department dominated by European historians.
> As you note, non-peer-reviewed materials on the web might be considered
> as a contribution to scholarship or teaching, but there are strong
> objections to doing so since such materials are very different from
> traditional scholarship and teaching. Perhaps the creation of online
> materials might be considered as service (a common third category of
> evaluation for faculty).
But sometimes services (if you mean what I think you mean, committees and
professional society activities, e.g.) don't have a scholarly component at
all, while a good web resource will. Perhaps a fourth category is waiting?
Patrick T. Rourke
ptrourke@mediaone.net
-------------------------------------------------------------------------
Humanist Discussion Group
Information at <http://www.kcl.ac.uk/humanities/cch/humanist/>
<http://www.princeton.edu/~mccarty/humanist/>
=========================================================================
This archive was generated by hypermail 2b29 : Thu Apr 13 2000 - 14:36:50 CUT