Active learning (AL) is built on a simple, compelling promise: if a model can choose which data it learns from, it can achieve higher accuracy with significantly fewer labeled examples. Traditionally, this happens in a "loop": a model is trained on a small labeled set, it identifies the most "informative" unlabeled points, a human labels them, and the model is retrained from scratch.
While this works for small models, the rise of deep learning has turned this loop into a massive computational bottleneck. As noted by Manabu Sassano, the overhead of retraining after every labeling iteration limits practical applications. When models have millions of parameters and datasets have millions of points, the "train-select-train" cycle becomes prohibitively expensive.
Recent research focuses on breaking this cycle, moving away from naive retraining toward strategies that prioritize "compute-efficiency" alongside "label-efficiency."
The Cost of Iteration
The primary friction in active learning is the frequency of retraining. In a standard setup, if you want to label 1,000 samples in batches of 10, you must retrain your model 100 times. For large-scale architectures, this is often impossible. Hyokun Yun and colleagues point out that this iterative process creates a significant overhead that makes AL impractical for complex architectures like those used in Named Entity Recognition.
To solve this, researchers are exploring three main avenues: using cheaper "surrogate" models for selection, moving to "one-shot" selection, and leveraging the power of pre-trained representations.
Surrogate Models and Parameter Efficiency
One way to lower the cost of the AL loop is to avoid using the full, heavy model during the data selection phase. Instead, we can use a "proxy" to decide what is informative.
Daria Galimzianova and Leonid Sanochkin demonstrate that adapters—small, parameter-efficient modules—can act as these proxies. By training only the adapters during the selection phase, they achieve the speed of a small model while maintaining the performance of the large one, reducing total compute time by over 20%.
Taking this further, Hasnain Irshad Bhatti and colleagues proposed PruneFuse. This method uses a pruned, "surrogate" version of the network to select data. Crucially, they don't just throw the surrogate away; they fuse its learned weights back into the original large model to provide a "warm start," accelerating the final training stage.
One-Shot and Training-Free Selection
If the loop itself is the problem, the most radical solution is to eliminate it.
One-shot active learning attempts to select the entire budget of informative samples in a single pass, before any retraining occurs. Sheng-Jun Huang and his team developed a method based on Lewis weight sampling. By treating the unlabeled data as a pool to be compressed, they can identify a single, highly informative batch that works across diverse model architectures, bypassing the need for hundreds of retraining cycles.
Alternatively, some methods aim to be training-free. Apivich Hemachandra and colleagues introduced EV-GP, which uses Neural Tangent Kernel (NTK) theory to select data points based on their expected variance. Because this relies on the mathematical properties of the network's initialization rather than its trained weights, it can select data without the "train-select-train" cycle, providing theoretical guarantees for both accuracy and stability.
Leveraging Pre-trained Representations
In the modern "pretrain-then-finetune" paradigm, we rarely start from scratch. This changes the goal of active learning: instead of building a representation, we are trying to adapt an existing one.
Jinghui Lu found that the choice of representation is a "make or break" factor for AL efficiency. Using powerful pre-trained models like BERT as the backbone for selection significantly reduces the number of manual labels needed compared to traditional methods like Bag-of-Words.
This leads to the concept of Active Finetuning. Yichen Xie and Han Lu argue that when you have a massive unlabeled dataset and a tiny budget (e.g., 1%), the challenge is identifying which samples will best "unlock" the knowledge already inside a pre-trained model. By focusing on the specific needs of the finetuning process rather than general uncertainty, these methods make AL viable for the massive scales of modern computer vision and NLP.
Conclusion
The evolution of active learning reflects a shift in priorities. Early work focused purely on reducing the number of labels. Today, researchers like Yichen Xie and Sheng-Jun Huang recognize that a label saved is not worth a week of GPU time. By using parameter-efficient modules, one-shot sampling, and pre-trained foundations, the field is moving toward a version of active learning that is as computationally efficient as it is data-efficient.