Abstract
Developed to alleviate prohibitive labeling costs, active learning (AL) methods aim to reduce label complexity in supervised learning. While recent work has demonstrated the benefit of using AL in combination with large pre-trained language models (PLMs), it has often overlooked the practical challenges that hinder the effectiveness of AL. We address these challenges by leveraging representation smoothness analysis to ensure AL is feasible, that is, both effective and practicable. Firstly, we propose an early stopping technique that does not require a validation set -- often unavailable in realistic AL conditions -- and observe significant improvements over random sampling across multiple datasets and AL methods. Further, we find that task adaptation improves AL, whereas standard short fine-tuning in AL does not provide improvements over random sampling. Our work demonstrates the usefulness of representation smoothness analysis for AL and introduces an AL stopping criterion that reduces label complexity.
Introduction
Active learning (AL) is designed to minimize labeling costs by selecting the most informative data points for human annotation. However, its practical application often encounters a significant hurdle: the lack of labeled validation sets. In standard supervised learning, validation sets are essential for early stopping to prevent overfitting. In the low-resource scenarios where AL is most valuable, dedicating a portion of the limited budget to a validation set is often a "Catch-22"—if you have enough labels for validation, the need for AL is diminished.
Furthermore, standard short fine-tuning regimes often result in AL performing no better than simple random sampling. This paper addresses these challenges by investigating the internal geometry of a model's representations. By asking whether representation smoothness can signal when a model has finished learning or when an unlabeled pool is depleted of information, the authors introduce a framework to make AL both effective and practicable.
Methodology: Representation Smoothness Analysis
The core of this work is the application of Besov Space theory, a mathematical framework for analyzing function smoothness, to the internal layers of Pre-trained Language Models (PLMs) like BERT.
BEAST (Besov Early STOpping)
BEAST monitors the distribution of representation smoothness across model layers to determine the optimal stopping point for training. Unlike traditional methods that require a validation loss curve, BEAST identifies overfitting by observing changes in the smoothness profile. As a model learns effectively, smoothness tends to peak in specific layers; conversely, when the model begins to memorize noise (overfit), this distribution flattens.
ALSBI (Active Learning Stopping by Besov Index)
To determine when to stop the entire active learning cycle, the authors propose ALSBI. This metric compares the smoothness of samples selected by the AL strategy to those selected randomly. When the smoothness profiles of "informative" samples align with random samples, it indicates "information depletion," signaling that further labeling will not yield significant gains.
RG (Representation Gradients)
The authors also introduce a new sampling strategy, Representation Gradients (RG). This method selects instances with the largest gradient norms in the representation space, prioritizing data that most significantly "surprises" the model's internal state.
Experimental Results
The methods were evaluated across five text classification benchmarks (TREC-6, TREC-2, SUBJ, AGN-4, and AGN-2).
Key Findings
- Performance Gains: Using the ETA (Task-Adaptive Pre-training + Extended Training) regime combined with BEAST, AL outperformed random sampling in 22 out of 25 experimental cases.
- Stability: BEAST significantly reduced the variance of results across different runs, addressing the instability often associated with AL.
- Label Complexity Reduction: ALSBI successfully identified points of maximum label savings, allowing for high efficiency without exhaustive labeling.
- Training Regime Impact: A critical insight is that the training regime (e.g., TAPT) often has a greater impact on AL success than the choice of specific sampling strategy (Entropy, Core-set, etc.).
Discussion and Limitations
The study confirms that earlier layers in PLMs generally learn generalizable features, while deeper layers are more susceptible to "heuristic memorization" or shortcuts. By using smoothness as a proxy for learning, overfitting can be detected purely through the training data's impact on the model's internal state.
However, several limitations remain. The evaluation focused on 12-layer "base" models (BERT and ELECTRA); it remains to be seen if these patterns hold for massive LLMs or different architectures. Additionally, the scope was limited to single-text classification, and tasks like sequence tagging or generative modeling may exhibit different representation geometries. Finally, while BEAST eliminates the need for a validation set, the Besov smoothness calculation involves hyperparameters that may require tuning in different contexts.
Conclusion
This work demonstrates that representation smoothness analysis provides a powerful, label-free mechanism for managing active learning. By enabling early stopping and identifying information depletion without validation data, techniques like BEAST and ALSBI make active learning a viable tool for real-world, low-resource NLP applications.
Summary updated