Abstract: Technology-assisted review (TAR) refers to human-in-the-loop active learning workflows for finding relevant documents in large collections. These workflows often must meet a target for the proportion of relevant documents found (i.e. recall) while also holding down costs. A variety of heuristic stopping rules have been suggested for striking this tradeoff in particular settings, but none have been tested against a range of recall targets and tasks. We propose two new heuristic stopping rules, Quant and QuantCI based on model-based estimation techniques from survey research. We compare them against a range of proposed heuristics and find they are accurate at hitting a range of recall targets while substantially reducing review costs.
Introduction
In fields such as legal eDiscovery and systematic medical reviews, experts utilize active learning to identify relevant documents among millions of candidates. The primary objective is typically to reach a specific level of recall—the proportion of all relevant documents found. However, determining when to stop the review process is a critical bottleneck. Stopping too early results in missing vital information, while stopping too late leads to excessive costs from human reviewers examining irrelevant files.
Existing stopping rules generally fall into three categories: sample-based (accurate but expensive), interventional (disruptive to model efficiency), and heuristic (utilizing data already labeled during review). Most current heuristics are "target-agnostic," meaning they are hard-coded to stop based on model stability rather than specific user-defined recall goals. This research, originally presented at DocEng '21, addresses this gap by introducing target-aware heuristics that adapt to any recall target.
Methodology
We introduce two new stopping rules based on Model-Based Estimation:
- Quant: This rule treats the probabilities generated by the active learning model (Logistic Regression) as estimates of relevance. It employs a Ratio Estimator for recall, comparing the sum of predicted probabilities in the reviewed set against the sum of predicted probabilities for the entire collection. This approach helps cancel out systematic errors where the model might be consistently over- or under-confident.
- QuantCI: Recognizing that point estimates can be noisy, QuantCI adds a "safety buffer" using a Taylor series expansion (the Delta Method) to approximate the variance of the recall estimate. The process stops only when the lower bound of a 95% confidence interval meets the user's recall target.
These methods were evaluated using the RCV1-v2 benchmark dataset (800,000+ Reuters news stories). To ensure generalizability, we selected 45 categories across a grid of document prevalence (frequency) and classification difficulty.
Results and Discussion
The study finds that model-based estimation provides a highly accurate and cost-effective way to implement stopping rules without additional labeling costs.
As shown in the experimental results across various task types, target-agnostic rules like the Knee Method are optimized for high recall (approx. 70-80%). When a user requires a lower target (e.g., 10% recall for a preliminary investigation), the Knee Method overshoots significantly, costing 21.5x the optimal cost. In contrast, QuantCI accurately tracks lower targets, costing only 6.0x the optimal at the 10% level.
Key performance metrics include:
- Accuracy: QuantCI achieved the lowest Mean Square Error (MSE) across all targets (0.112 vs. 0.195 for the Knee Method).
- Reliability: QuantCI effectively adjusted its stopping point based on task difficulty, whereas older rules often stopped at inconsistent recall levels depending on whether a category was "Easy" or "Hard."
- Robustness: At high recall targets (70%), QuantCI remained statistically indistinguishable from the Knee Method, proving it is robust even in scenarios the Knee Method was specifically designed for.
Conclusion
Quant and QuantCI represent a significant advancement in TAR stopping rules by providing target-aware, model-based estimation. By leveraging the classifier already being trained in the workflow, these methods offer "free" estimates that allow for precise control over the recall-cost tradeoff. While challenges remain at extremely high recall targets (90%+) due to model calibration issues, these heuristics offer substantial practical benefits for real-world document review tasks. Future work will involve testing these methods on messy, real-world eDiscovery production data.
Summary updated