Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review
Richmond Journal of Law & Technology, Vol. 17, Issue 3, Article 11 (2011)
The paper that directly tested—and rejected—the assumption that reviewing every document by hand is necessarily the most reliable way to conduct large-scale legal discovery.
Overview
Using data from the TREC 2009 Legal Track, the article compares technology-assisted review with the results that would have been obtained through exhaustive manual review by the official assessors. It shows that a process combining human judgment with automated ranking can be not only dramatically more efficient, but also more effective as measured by recall, precision, and F1.
Key contributions
- Placed manual review and technology-assisted review on the same empirical footing.
- Used recall, precision, and F1 rather than intuition or reviewer effort as measures of effectiveness.
- Analyzed real TREC Legal Track data and the work of two distinct technology-assisted review teams.
- Documented substantial inconsistency and error in human relevance assessment.
- Helped establish the scientific foundation for judicial and professional acceptance of TAR.
Abstract
E-discovery processes that use automated tools to prioritize and select documents for review are typically regarded as potential cost-savers—but inferior alternatives—to exhaustive manual review, in which a cadre of reviewers assesses every document for responsiveness to a production request, and for privilege. This Article offers evidence that such technology-assisted processes, while indeed more efficient, can also yield results superior to those of exhaustive manual review, as measured by recall and precision, as well as F1, a summary measure combining both recall and precision. The evidence derives from an analysis of data collected from the TREC 2009 Legal Track Interactive Task, and shows that, at TREC 2009, technology-assisted review processes enabled two participating teams to achieve results superior to those that could have been achieved through a manual review of the entire document collection by the official TREC assessors.
Citation
Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, 17 Rich. J.L. & Tech. 11 (2011).