Cross evaluation - A pilot application of a new evaluation mechanism

Ying Sun, Paul Kantor, Tomek Strzalkowski, Robert Rittman, Faye Wacholder

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

The work reports some initial success in extending the Rutgers Paradigm of IR evaluation to the realm of concrete measurement, not in information retrieval per se, but in the arguably more complex domain of Question Answering. Crucial to the paradigm are two components: cross evaluation, and an analytical model that controls for the potential problems of cross evaluation. We describe the experimental design and analytical models. In the models, interaction effects are examined and found not to be important. After eliminating the interaction effects, we are able to extract meaningful and useful results from a very small study involving just three analysts, five topics, and two "systems".

Original languageEnglish (US)
Pages (from-to)383-392
Number of pages10
JournalProceedings of the ASIST Annual Meeting
Volume41
DOIs
StatePublished - Nov 1 2004

Fingerprint

Analytical models
Information retrieval
evaluation
Design of experiments
paradigm
Concretes
interaction
information retrieval

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Library and Information Sciences

Cite this

Sun, Ying ; Kantor, Paul ; Strzalkowski, Tomek ; Rittman, Robert ; Wacholder, Faye. / Cross evaluation - A pilot application of a new evaluation mechanism. In: Proceedings of the ASIST Annual Meeting. 2004 ; Vol. 41. pp. 383-392.
@article{27548d4feaa448b0991b06b02844ab38,
title = "Cross evaluation - A pilot application of a new evaluation mechanism",
abstract = "The work reports some initial success in extending the Rutgers Paradigm of IR evaluation to the realm of concrete measurement, not in information retrieval per se, but in the arguably more complex domain of Question Answering. Crucial to the paradigm are two components: cross evaluation, and an analytical model that controls for the potential problems of cross evaluation. We describe the experimental design and analytical models. In the models, interaction effects are examined and found not to be important. After eliminating the interaction effects, we are able to extract meaningful and useful results from a very small study involving just three analysts, five topics, and two {"}systems{"}.",
author = "Ying Sun and Paul Kantor and Tomek Strzalkowski and Robert Rittman and Faye Wacholder",
year = "2004",
month = "11",
day = "1",
doi = "10.1002/meet.1450410145",
language = "English (US)",
volume = "41",
pages = "383--392",
journal = "Proceedings of the ASIST Annual Meeting",
issn = "1550-8390",
publisher = "Learned Information",

}

Cross evaluation - A pilot application of a new evaluation mechanism. / Sun, Ying; Kantor, Paul; Strzalkowski, Tomek; Rittman, Robert; Wacholder, Faye.

In: Proceedings of the ASIST Annual Meeting, Vol. 41, 01.11.2004, p. 383-392.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Cross evaluation - A pilot application of a new evaluation mechanism

AU - Sun, Ying

AU - Kantor, Paul

AU - Strzalkowski, Tomek

AU - Rittman, Robert

AU - Wacholder, Faye

PY - 2004/11/1

Y1 - 2004/11/1

N2 - The work reports some initial success in extending the Rutgers Paradigm of IR evaluation to the realm of concrete measurement, not in information retrieval per se, but in the arguably more complex domain of Question Answering. Crucial to the paradigm are two components: cross evaluation, and an analytical model that controls for the potential problems of cross evaluation. We describe the experimental design and analytical models. In the models, interaction effects are examined and found not to be important. After eliminating the interaction effects, we are able to extract meaningful and useful results from a very small study involving just three analysts, five topics, and two "systems".

AB - The work reports some initial success in extending the Rutgers Paradigm of IR evaluation to the realm of concrete measurement, not in information retrieval per se, but in the arguably more complex domain of Question Answering. Crucial to the paradigm are two components: cross evaluation, and an analytical model that controls for the potential problems of cross evaluation. We describe the experimental design and analytical models. In the models, interaction effects are examined and found not to be important. After eliminating the interaction effects, we are able to extract meaningful and useful results from a very small study involving just three analysts, five topics, and two "systems".

UR - http://www.scopus.com/inward/record.url?scp=33645022946&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33645022946&partnerID=8YFLogxK

U2 - 10.1002/meet.1450410145

DO - 10.1002/meet.1450410145

M3 - Article

AN - SCOPUS:33645022946

VL - 41

SP - 383

EP - 392

JO - Proceedings of the ASIST Annual Meeting

JF - Proceedings of the ASIST Annual Meeting

SN - 1550-8390

ER -