Selected publications / Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review

Continuous active learning · High recall

Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review

Gordon V. Cormack and Maura R. Grossman

arXiv:1504.06868 (2015)

A design for continuous active learning that minimizes topic-specific tuning and makes high-recall review more autonomous and dependable.

PDF arXiv Google Scholar BibTeX Citation

Overview

This paper develops “Auto TAR,” a continuous active learning process intended to require very little specialized search engineering. The reviewer begins with a short query, a topic description, or a single relevant document; the system then repeatedly learns from the reviewer’s relevance judgments and selects the next documents to examine.

The central design goal is not merely strong average performance. In high-stakes review, a method must work reliably on the particular matter at hand. The experiments therefore examine both effectiveness and the frequency and visibility of failures across legal, news, ad-hoc retrieval, and filtering tasks.

Why this paper matters. The paper moves CAL from an effective research protocol toward a practical, nearly turnkey procedure. It also makes reliability—not just average accuracy—a first-class criterion for high-recall systems.

Key contributions

Eliminates topic- and dataset-specific tuning choices from the review loop.
Shows how a single relevant or synthetic seed document can start the process.
Uses temporary presumptive non-relevant examples to train from a minimal seed.
Tests generalization across four distinct families of retrieval tasks.

Suggested citation

Gordon V. Cormack & Maura R. Grossman, Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review, arXiv:1504.06868 (2015).