Maura R. GrossmanSelected publications
Selected publications / Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me
Human judgment · Evaluation · Total recall

Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me

Gordon V. Cormack and Maura R. Grossman

Proceedings of SIGIR 2017, pp. 5–14

Explores how disagreement and uncertainty in human relevance assessments affect the evaluation of high-recall review methods.

DOIIR AnthologyGoogle ScholarBibTeXCitation

Overview

Technology-assisted review aims to approach total recall and precision while minimizing human effort. But its training data and evaluation labels come from people, whose judgments are variable and imperfect. This paper treats human assessments as indirect evidence of the underlying—and sometimes ambiguous—property of relevance.

The memorable case study concerns 401,960 email messages originally reviewed by a senior state records archivist, Roger. A later blind adjudication suggested that TAR could have achieved comparable recall and better precision while requiring review of substantially fewer messages.

Why this paper matters. A particular assessor’s judgments are evidence, not an infallible gold standard. Different qualified assessors may reasonably disagree, and even the same assessor may not always make identical judgments. Sound evaluation should therefore compare review effectiveness in spite of this unavoidable uncertainty, rather than reward a system for reproducing the random idiosyncrasies of one assessor.

Key contributions

  • Treats human judgments as fallible observations rather than absolute truth.
  • Connects judgment error to both TAR evaluation and relevance-feedback simulation.
  • Tests the argument on TREC data and a real archival email collection.
  • Shows how hybrid human–computer review can improve both accuracy and efficiency.

Suggested citation

Gordon V. Cormack & Maura R. Grossman, Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me, in Proceedings of the 40th International ACM SIGIR Conference 5–14 (2017), doi:10.1145/3077136.3080812.