Abstract
Technology-assisted review (TAR) refers to human-in-the-loop machine learning workflows for document review in legal discovery and other high recall review tasks. Attorneys and legal technologists have debated whether review should be a single iterative process (one-phase TAR workflows) or whether model training and review should be separate (two-phase TAR workflows), with implications for the choice of active learning algorithm. The relative cost of manual labeling for different purposes (training vs. review) and of different documents (positive vs. negative examples) is a key and neglected factor in this debate. Using a novel cost dynamics analysis, we show analytically and empirically that these relative costs strongly impact whether a one-phase or two-phase workflow minimizes cost. We also show how category prevalence, classification task difficulty, and collection size impact the optimal choice not only of workflow type, but of active learning method and stopping point.
Introduction: The TAR Debate
In legal "eDiscovery," professionals must identify relevant documents within massive datasets. This process, known as Technology-Assisted Review (TAR), relies on active learning to prioritize review. For years, the industry has debated two primary workflows:
- One-phase TAR (TAR 2.0): An iterative model where every reviewed document contributes to both training and the final production set.
- Two-phase TAR (TAR 1.0): A distinct training phase (Phase 1) followed by a separate, high-volume bulk review phase (Phase 2) using the finalized model.
While many practitioners claim one-phase workflows are inherently superior, this research demonstrates that such claims often ignore the reality of asymmetric costs—specifically, the difference in rates between senior attorneys (training) and contract reviewers (bulk review).
The Four-Tuple Cost Model
To resolve this debate, we propose a formal mathematical framework using a four-tuple cost model $s = (\alpha_p, \alpha_n, \beta_p, \beta_n)$. Here, $\alpha$ represents the cost of review during training (Phase 1), while $\beta$ represents the cost during bulk review (Phase 2). The subscripts $p$ and $n$ denote the costs for positive (relevant) and negative (irrelevant) documents, respectively.
As shown in the Cost Dynamics Graphs above, the total cost is a combination of training effort and the "failure penalty" of documents left for Phase 2. The dashed vertical line indicates the optimal stopping point where total cost is minimized. When costs are uniform (Figure 1a), the dynamics differ significantly from scenarios where training is more expensive than bulk review (Figures 1b and 1c).
Active Learning Strategies and Task Difficulty
The study evaluated two primary active learning strategies across standard benchmarks like RCV1-v2 and the Jeb Bush Email Collection:
- Relevance Feedback (Exploitation): Prioritizes documents the model believes are most likely relevant. This is highly effective for "Easy" tasks where relevant documents are rare.
- Uncertainty Sampling (Exploration): Targets documents where the model is least certain to refine the classification boundary. This is essential for "Hard" tasks to ensure the model is robust enough for Phase 2.
Key Findings
Our analysis reveals several critical insights for legal technologists:
- The Amortization of Training: In large datasets, Uncertainty Sampling in a two-phase workflow often pays for itself. While it may uncover more "useless" negatives during Phase 1, the resulting model accuracy can reduce bulk review costs by up to 60%.
- One-Phase Limitations: One-phase workflows are only optimal under uniform cost conditions—where a senior partner's time costs the same as a contract reviewer's, and training documents are as easy to review as bulk documents.
- Workflow Dominance: In 100% of the "Common" category scenarios in the RCV1-v2 dataset, uncertainty sampling in a two-phase workflow was the most efficient choice.
Conclusion
The choice between TAR 1.0 and TAR 2.0 should not be a matter of dogma but a calculation based on cost structure and task difficulty. By identifying the deterministic "ideal" stopping point for training, legal teams can significantly reduce the total cost of discovery, particularly when training costs are high relative to bulk review rates.
Summary updated