Good Data from Bad Models : Foundations of Threshold-based Auto-labeling

Harit Vishwakarma, Heguang Lin, Frederic Sala, Ramya Korlakai Vinayak

Abstract

Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Auto-labeling systems are a promising way to reduce reliance on manual labeling for dataset construction. Threshold-based auto-labeling, where validation data obtained from humans is used to find a threshold for confidence above which the data is machine-labeled, is emerging as a popular solution used widely in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. In this work, we analyze threshold-based auto-labeling systems and derive sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of machine-labeled data. Our results provide two insights. First, reasonable chunks of the unlabeled data can be automatically and accurately labeled by seemingly bad models. Second, a hidden downside of threshold-based auto-labeling systems is potentially prohibitive validation data usage. Together, these insights describe the promise and pitfalls of using such systems. We validate our theoretical guarantees with simulations and study the efficacy of threshold-based auto-labeling on real datasets.

Introduction

The primary bottleneck in modern supervised machine learning is the acquisition of high-quality labeled data. While manual labeling remains the gold standard, its cost is often prohibitive at scale. Threshold-based Auto-labeling (TBAL) has emerged as a popular industry solution, where a system only accepts a model's prediction if its confidence score exceeds a specific threshold. This paper provides the first formal theoretical characterization of TBAL, revealing that while it can generate high-quality datasets from "bad" models, it incurs a significant "validation tax" to guarantee label quality.

Methodology: The TBAL Framework

The authors provide a formal theoretical characterization of TBAL, analyzing an iterative algorithm designed to balance machine efficiency with human oversight. The process operates in a continuous loop:

  1. Train: An initial model is trained on a small "seed" set of human-labeled data.
  2. Estimate: Using a separate validation set, the system identifies a confidence threshold where the model’s error rate is guaranteed to be below a user-defined target ($\epsilon_a$).
  3. Auto-label: Unlabeled points with confidence scores above this threshold are automatically added to the dataset.
  4. Active Query: Points falling in uncertain regions are sent to human labelers (Active Learning) to refine the model for subsequent rounds.
The iterative loop of Threshold-based Auto-labeling (TBAL) architecture.

The core contribution is the derivation of sample complexity bounds using Rademacher complexity. These bounds quantify the amount of validation data required to maintain the auto-labeling error within specified limits with high probability.

Results and Insights

The research demonstrates a counter-intuitive phenomenon: "Good Data from Bad Models." A model with a high global error rate (e.g., a linear classifier on non-linear data) can still be highly accurate on specific high-confidence sub-regions. TBAL exploits these "pockets of accuracy" to build high-quality datasets using imperfect models.

Comparison of TBAL performance and coverage on the Circles dataset, showing how it outperforms standard Active Learning (AL) baselines.

Key Findings:

  • The Circles Success: In experiments with concentric circles, a simple linear model—fundamentally ill-suited for the geometry—successfully auto-labeled nearly 100% of the data with less than 1% error by iteratively carving away high-confidence regions.
  • The Validation Tax: Guaranteeing quality is expensive. The theory shows that validation data requirements scale with $O(1/\epsilon_a^2)$. If the validation set is too small, the system becomes overconfident, leading to error spikes that exceed the target threshold.
  • Coverage Gains: TBAL consistently achieves higher coverage (percentage of dataset labeled) than standard Active Learning baselines while adhering to strict error constraints across synthetic and real-world benchmarks like Tiny-ImageNet and IMDB Reviews.

Conclusion and Future Directions

While TBAL is powerful, its efficacy is tied to model calibration. If a model is "confidently wrong," the system fails to maintain quality. Future work is needed to address the high "validation tax" through methods like Active Testing and to incorporate "noisy" human labels into the theoretical bounds. Ultimately, this work clarifies that while Active Learning seeks the best model, Auto-labeling seeks the best dataset, and TBAL provides a rigorous path toward the latter.

Summary updated

Abstract

Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Auto-labeling systems are a promising way to reduce reliance on manual labeling for dataset construction. Threshold-based auto-labeling, where validation data obtained from humans is used to find a threshold for confidence above which the data is machine-labeled, is emerging as a popular solution used widely in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. In this work, we analyze threshold-based auto-labeling systems and derive sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of machine-labeled data. Our results provide two insights. First, reasonable chunks of the unlabeled data can be automatically and accurately labeled by seemingly bad models. Second, a hidden downside of threshold-based auto-labeling systems is potentially prohibitive validation data usage. Together, these insights describe the promise and pitfalls of using such systems. We validate our theoretical guarantees with simulations and study the efficacy of threshold-based auto-labeling on real datasets.
BibTeX
@inproceedings{vishwakarma2023good,
  author = {Harit Vishwakarma and Heguang Lin and Frederic Sala and Ramya Korlakai Vinayak},
  title = {{Good Data from Bad Models : Foundations of Threshold-based Auto-labeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=RUCFAKNDb2},
}
Figure