Information retrieval · Validation · SIGIR 2024

Unbiased Validation of Technology-Assisted Review for eDiscovery

Gordon V. Cormack, Maura R. Grossman, Andrew Harbison, Tom O’Halloran, and Bronagh McManus

Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2677–2681 (2024)

A practical method for comparing technology-assisted review with exhaustive manual review without quietly building the answer into the validation sample.

Read PDF Publication record DOI BibTeX Plain citation

Authors: Gordon V. Cormack, Maura R. Grossman, Andrew Harbison, Tom O’Halloran, and Bronagh McManus
Published: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2677–2681 (2024)
Format: Article / scholarly paper

Overview

Recall estimates are meaningful only when the relevance assessments used for validation are independent of the process being evaluated. In litigation, that condition is often violated: the same review decisions help train, stop, and validate the system. This paper presents two strategies that embed blinded relevance assessment within a TAR workflow so that the effectiveness of TAR and exhaustive manual review can be compared without that circularity.

Central contribution. Validation is often treated as a final sampling exercise, but a sample drawn or judged in a way that is entangled with the review process can systematically favour one method. The paper moves the question from “What recall number did the project report?” to “Was the comparison designed so that either method could fairly win?” It also demonstrates the approach in a large real-world review arising from litigation over the accounting practices preceding the collapse of a major insurer.

Key contributions

Explains why recall estimates based on non-independent judgments may be biased or misinterpreted.
Presents two blinded validation strategies suitable for an operational TAR project.
Supports a direct comparison between TAR and the counterfactual of exhaustive manual review.
Uses a large-scale, real-world legal review rather than a laboratory-only collection.
Shows how validation can be designed into the review instead of appended after the fact.

Paper summary

It is well established that recall estimation is valid only when based on independent relevance assessments, and is most useful for comparing the relative effectiveness of competing methods. Those conditions are rarely satisfied when validating discovery efforts in litigation. The paper presents two unbiased strategies that embed blinded relevance assessment in a TAR process, and illustrates them in litigation concerning the accounting practices that preceded the collapse of a major insurance company.

Citation

Gordon V. Cormack, Maura R. Grossman, Andrew Harbison, Tom O’Halloran & Bronagh McManus, Unbiased Validation of Technology-Assisted Review for eDiscovery, in Proceedings of SIGIR ’24, 2677–2681 (2024). https://doi.org/10.1145/3626772.3657903

Unbiased Validation of Technology-Assisted Review for eDiscovery

Overview

Key contributions

Paper summary

Citation

Related publications