On minimizing cost in legal document review workflows

Eugene Yang, David D. Lewis, Ophir Frieder

TL;DR

This paper introduces a formal mathematical framework and a novel visualization tool—the cost dynamics graph—to resolve the long-standing debate in legal tech over whether "one-phase" or "two-phase" document review workflows are superior. The authors demonstrate that the optimal choice depends entirely on cost asymmetries (e.g., senior vs. junior attorney rates) and task characteristics, proving that the industry trend toward one-phase workflows is only economically sound under very specific, uniform cost conditions.

What Problem Does This Paper Address?

In legal "eDiscovery," lawyers must sift through millions of documents to find those relevant to a case. This is done using Technology-Assisted Review (TAR), where machine learning models (active learning) help prioritize documents.

There is a heated debate between:

  1. One-phase TAR (TAR 2.0): An iterative process where every document reviewed is used for both training the model and as part of the final production.
  2. Two-phase TAR (TAR 1.0): A training phase (Phase 1) followed by a separate, high-volume review phase (Phase 2) using the finalized model.

Proponents of one-phase workflows often claim they are inherently superior. However, this paper argues that previous comparisons ignored a crucial reality: costs are not uniform. Reviewing a document for training (often done by expensive senior associates) costs more than bulk review (often done by contract attorneys). Furthermore, confirming a "positive" (relevant) document often takes longer than dismissing a "negative" one. The authors set out to model how these varying costs dictate the most efficient workflow.

Methods

The authors propose a four-tuple cost model $s = (\alpha_p, \alpha_n, \beta_p, \beta_n)$, where:

  • $\alpha$: Cost of review in Phase 1 (training).
  • $\beta$: Cost of review in Phase 2 (bulk review).
  • $p/n$: Subscripts for positive vs. negative document review costs.

They also introduce the Cost Dynamics Graph, a stacked area chart that tracks four cost components across iterations of active learning. This allows researchers to visualize the "stopping point" where the total cost (training cost + the "failure penalty" of documents left for Phase 2) is minimized.

The study compares two primary active learning strategies:

  • Relevance Feedback (Exploitation): Reviewing documents the model thinks are most likely relevant.
  • Uncertainty Sampling (Exploration): Reviewing documents the model is most unsure about to improve the classifier's boundary.

Datasets

The researchers used two standard benchmarks for high-recall retrieval:

  • RCV1-v2: A collection of ~804,000 Reuters news articles categorized by topic. They tested both the full set and a 20% subset to study collection size effects.
  • Jeb Bush Email Collection: ~274,000 emails from the former Florida Governor’s administration, providing a realistic "messy" legal dataset.

Tasks were binned by prevalence (how many relevant docs exist) and difficulty (how hard they are for a logistic regression model to classify).

Key Insights and Ideas

  • The "Amortization" of Training: In large datasets, it often pays to use "Uncertainty Sampling" in a two-phase workflow. Even though it uncovers many "useless" negatives during training (increasing Phase 1 costs), the resulting model is so much more accurate that it saves massive amounts of time/money in the high-volume Phase 2.
  • One-Phase is an Artifact of Uniform Cost: One-phase workflows are only optimal when the cost of a senior partner's time equals a contract reviewer's time, and when training documents are just as easy to review as bulk documents.
  • Task Difficulty Matters: For "Easy" tasks with rare relevant documents, Relevance Feedback is king because you can find the "needle in the haystack" quickly. For "Hard" tasks, Uncertainty Sampling is necessary to build a model that doesn't collapse during bulk review.
  • The Myth of "TAR 2.0" Superiority: The paper analytically proves that two-phase workflows can be significantly cheaper (up to 60% reduction) when training costs are high relative to review costs.

Key Results

  • Cost Reductions: Under asymmetric cost structures (e.g., training is 10x more expensive than review), two-phase workflows consistently outperformed one-phase workflows.
  • Workflow Dominance: In 100% of the RCV1-v2 "Common" category scenarios, uncertainty sampling in a two-phase workflow was the most efficient choice.
  • Statistical Significance: Using the Kolmogorov–Smirnov test, the authors showed that the cost differences between these workflows are statistically significant across hundreds of category/seed pairs.
  • Stopping Points: The model provides a deterministic way to identify the "ideal" stopping point for training, showing that the optimal time to switch to bulk review happens much earlier when training is expensive.

Limitations and Future Work

  • Idealized Stopping: The model assumes we know when we've hit the optimal stopping point. In the real world, lawyers have to estimate their recall, which adds its own cost and margin of error.
  • Deterministic Costs: The model assumes fixed costs per document type, whereas real-world review speeds vary by individual reviewer and document complexity.
  • Recall Target: The study focused on a 0.80 recall target. Future work could explore how these dynamics shift for extremely high-recall requirements (e.g., 0.95+), which are common in high-stakes litigation.

Summary updated

Abstract

Technology-assisted review (TAR) refers to human-in-the-loop machine learning workflows for document review in legal discovery and other high recall review tasks. Attorneys and legal technologists have debated whether review should be a single iterative process (one-phase TAR workflows) or whether model training and review should be separate (two-phase TAR workflows), with implications for the choice of active learning algorithm. The relative cost of manual labeling for different purposes (training vs. review) and of different documents (positive vs. negative examples) is a key and neglected factor in this debate. Using a novel cost dynamics analysis, we show analytically and empirically that these relative costs strongly impact whether a one-phase or two-phase workflow minimizes cost. We also show how category prevalence, classification task difficulty, and collection size impact the optimal choice not only of workflow type, but of active learning method and stopping point.
BibTeX
@inproceedings{yang2021minimizing,
  author = {Eugene Yang and David D. Lewis and Ophir Frieder},
  title = {{On minimizing cost in legal document review workflows}},
  booktitle = {DocEng},
  year = {2021},
  url = {https://openreview.net/forum?id=EUUdM23FWp},
}

Loading related content...

Figure