Selected publications / Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me

Human judgment · Evaluation · Total recall

Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me

Gordon V. Cormack and Maura R. Grossman

Proceedings of SIGIR 2017, pp. 5–14

A challenge to the convenient fiction that human relevance labels are perfect ground truth.

DOI IR Anthology Google Scholar BibTeX Citation

Overview

Technology-assisted review aims to approach total recall and precision while minimizing human effort. But its training data and evaluation labels come from people, whose judgments are variable and imperfect. This paper models human assessments as indirect evidence of the underlying—and sometimes ambiguous—property of relevance.

The memorable case study concerns 401,960 email messages originally reviewed by a senior state records archivist, Roger. A later blind adjudication suggested that TAR could have achieved comparable recall and better precision while requiring review of substantially fewer messages.

Why this paper matters. The paper changes how apparent machine “errors” should be interpreted. When a system disagrees with a prior reviewer, the system may have exposed a missed relevant document rather than made a mistake. That insight matters for evaluation, training, quality control, and claims about a supposed ceiling imposed by human agreement.

Key contributions

Models human judgments as fallible observations rather than absolute truth.
Connects judgment error to both TAR evaluation and relevance-feedback simulation.
Tests the argument on TREC data and a real archival email collection.
Shows how hybrid human–computer review can improve both accuracy and efficiency.

Suggested citation

Gordon V. Cormack & Maura R. Grossman, Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me, in Proceedings of the 40th International ACM SIGIR Conference 5–14 (2017), doi:10.1145/3077136.3080812.