In machine learning, the "more is better" philosophy of data collection eventually hits a wall of economic reality. Whether it is a pathologist identifying mitotic figures in cancer biopsies or a linguist disambiguating word senses, human expertise is expensive, slow, and finite.
The central challenge in modern data-centric AI is not just how to label data, but knowing when to stop. Research across diverse domains—from medical imaging to natural language understanding—has converged on a shared goal: identifying the saturation point where the marginal utility of an additional human label no longer justifies its cost.
The Geometry of the Learning Curve
At the heart of this research is the behavior of the learning curve. As Jiang Bian, Xi Yang, and Yonghui Wu demonstrate in Customize Deep Learning-based De-Identification Systems Using Local Clinical Notes, model performance typically follows a power law. Initial data points provide massive leaps in accuracy, but these gains quickly plateau.
The difficulty lies in the fact that we usually don't know the shape of this curve until we have already spent the budget. To solve this, Ernie Chang and colleagues in Revisiting Sample Size Determination in Natural Language Understanding propose modeling these curves using as little as 10% of the total data. By extrapolating the curve, practitioners can forecast the maximum achievable performance and detect early on if a dataset is of poor quality or if the model has already reached its "economic saturation point."
Automated Stopping Criteria
If we can't see the whole curve, how do we decide to stop in real-time? Researchers have proposed several "triggers" to terminate the labeling loop:
- Uncertainty and Entropy Stability: In Improving Classification Performance With Human Feedback, Natan Vidra and Thomas Clifford show that model entropy—a measure of a model's "confusion"—stabilizes after a relatively small number of expert-annotated samples. When new labels stop reducing the model's overall uncertainty, the human can be taken out of the loop.
- Label Consensus: In large-scale image classification, Yuan-Hong Liao and Sanja Fidler suggest an early-stop criterion based on the stability of the inferred label set in Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets. If additional human queries are unlikely to change the consensus on an image, further annotation is a waste of resources.
- Economic Cost-Benefit: With the rise of Large Language Models (LLMs), the "cost" is no longer just human time but also API tokens. Xiang Chen and Xintong Li note in From Selection to Generation: A Survey of LLM-based Active Learning that stopping criteria must now integrate monetary costs directly into the logic, terminating when the financial cost of the next token outweighs the predicted accuracy gain.
Strategic Sampling to Reach the Plateau Faster
If the goal is to reach the plateau with the least effort, the order of labeling matters as much as the amount. This is the core of Active Learning (AL).
Traditional AL focuses on "hard" examples, but Roi Reichart and Ari Rappoport argue in Sample Selection for Statistical Parsers that we must also consider "cognitive load." A sentence might be informative for a model but nearly impossible for a human to parse correctly. By selecting samples that are both informative and cognitively manageable, we reach the saturation point more efficiently.
Similarly, in structured learning, Hangfeng He and Qiang Ning found that "partial annotation" is often superior to complete annotation. In Partial Or Complete, That's The Question, they show that the information gain from the final few labels in a complex structure (like a full sentence parse) is much lower than the first few. Spreading a fixed budget across many partially labeled examples allows a model to see more diverse data before hitting the point of diminishing returns.
Why This Works: The Law of Redundancy
The reason these stopping points exist is that real-world data is highly redundant. In Corpus Expansion for Neural CWS on Microblog-Oriented Data, Jing Zhang and colleagues found that once a linguistic pattern has been seen a few times, additional exposures provide almost zero marginal utility.
By treating data acquisition as a dynamic optimization problem—using tools like Multi-Armed Bandits as suggested by Linqi Song in Stream-based Online Active Learning—we can treat every labeling decision as a trade-off between exploration (learning something new) and exploitation (using what we know). The "stopping point" is simply the moment the system realizes that the world has no more surprises left to offer for the current task.