Boosting Active Learning with Knowledge Transfer

Tianyang Wang, Xi Xiao, Gaofei Chen, Xiaoying Liao, Guo Cheng, Yingrui Ji

TL;DR

This paper introduces a novel active learning (AL) strategy that leverages knowledge transfer between a "teacher" and "student" model to improve uncertainty estimation for unlabeled data. By measuring the disagreement between these models, they can more effectively select informative samples for annotation, leading to better performance with less labeled data, particularly in challenging domains like cryo-electron tomography.

What Problem Does This Paper Address?

Supervised deep learning models excel when provided with vast amounts of labeled data. However, acquiring these labels can be prohibitively expensive, especially in specialized fields like computational biology (e.g., cryo-electron tomography for classification). Active learning (AL) aims to mitigate this by intelligently selecting the most informative unlabeled samples for human annotation, thereby maximizing model performance within a limited labeling budget.

Existing AL methods often struggle with estimating uncertainty effectively. Many rely on complex auxiliary models (like VAEs or GANs) or intricate training schemes, which are difficult to tune and adapt to new tasks. Furthermore, some methods incorrectly equate data uncertainty with the predicted task loss, a notion the authors argue is flawed. This paper seeks to address these limitations by proposing a simpler, more efficient, and broadly applicable method for uncertainty estimation in AL.

Methods

The core of this paper's contribution lies in its novel approach to uncertainty estimation within an active learning framework. They propose a teacher-student knowledge transfer setup.

  • Teacher-Student Architecture: The "teacher" model is the primary task model being trained. The "student" model shares the same architecture (or a slightly shallower version) and is trained simultaneously with the teacher.
  • Knowledge Transfer Loss: The student model is optimized using two loss functions:
  • Task Loss: The standard supervised loss for the given task (e.g., cross-entropy for classification).
  • Knowledge Transfer Loss: This loss encourages the student's feature representations to align with the teacher's. The paper specifically uses an "attention-transfer loss" over feature space, which they found to be effective.
  • Uncertainty Estimation: The key insight is that an unlabeled sample is considered uncertain if the teacher and student models disagree on its prediction. This disagreement is quantified using the KL divergence between the softmax outputs of the teacher and student models for classification tasks. For semantic segmentation, they use Mean Squared Error (MSE) on probability maps.
  • Data Selection: Samples with higher KL divergence (or MSE) are deemed more uncertain and are prioritized for annotation.
  • Theoretical Justification: The authors theoretically demonstrate that their selection strategy prioritizes data that leads to a higher upper-bound of the task loss, providing a principled explanation for why their method is effective.

This approach is designed to be task-agnostic, avoiding the need for specialized auxiliary networks or complex training procedures.

Datasets

The proposed method was evaluated on a diverse set of datasets:

  • Image Classification:
    • CIFAR-10
    • CIFAR-100
    • SVHN
    • Caltech101
    • ImageNet
  • Semantic Segmentation:
    • Cityscapes
  • Cryo-Electron Tomography (Cryo-ET):
    • A simulated cryo-ET dataset (50-snr005)
    • A real cryo-ET dataset (10c-real)

Key Insights and Ideas

  • Disagreement as Uncertainty: The most significant insight is that the disagreement between a task model (teacher) and a knowledge-transferring student model is a powerful indicator of data uncertainty. This elegantly sidesteps the need for complex, task-specific uncertainty estimation modules.
  • Task Loss vs. Upper-Bound of Task Loss: The paper challenges the common assumption that uncertainty is directly tied to the predicted task loss. Instead, they propose that uncertainty is more closely related to the upper-bound of the task loss, which their method effectively targets.
  • Simplicity and Generality: The teacher-student knowledge transfer framework is inherently simpler and more generalizable than many existing AL methods that rely on bespoke auxiliary architectures. This makes it adaptable to various domains and tasks without significant re-engineering.
  • Robustness to Noise: The method shows particular strength on noisy datasets, such as the simulated cryo-ET data, suggesting its ability to handle challenging real-world scenarios.

Key Results

The proposed method consistently outperformed state-of-the-art active learning baselines across various benchmarks:

  • Image Classification: Achieved higher accuracy on CIFAR-10, CIFAR-100, SVHN, Caltech101, and ImageNet, often surpassing results obtained by training on the full dataset. For example, on CIFAR-10, their method achieved 94.27% accuracy, exceeding the full dataset training result of 93.6%.
  • Semantic Segmentation: Demonstrated superior performance on the Cityscapes dataset, measured by mIoU.
  • Cryo-ET: Showed strong performance on both simulated and real cryo-ET datasets, outperforming other methods by a significant margin, especially on the noisy simulated data.
  • Generalization: The data selected by their method led to better generalization for a deeper task model (ResNet-50 trained on data selected by ResNet-18), indicated by a smaller gap between training and testing accuracy.
  • Efficiency: The method is computationally efficient due to its simpler architecture and training scheme.

Limitations and Future Work

  • Probability Calibration: While the paper uses KL divergence without explicit probability calibration, they acknowledge that modern neural networks are often not well-calibrated. They explore this in an ablation study, finding that probability calibration yields similar results, suggesting their method is robust to this issue. However, further investigation into optimal calibration strategies for their framework could be a future direction.
  • Task-Agnosticism: While the method is presented as task-agnostic, the specific choice of uncertainty metric (KL divergence for classification, MSE for segmentation) might require task-specific adaptation. Future work could explore more universally applicable uncertainty metrics.
  • Student Model Architecture: The paper uses a student model with the same number of modules as the teacher. Investigating the impact of different student model architectures or depths on knowledge transfer and uncertainty estimation could be a fruitful avenue.

Summary updated

Abstract

Uncertainty estimation is at the core of Active Learning (AL). Most existing methods resort to complex auxiliary models and advanced training fashions to estimate uncertainty for unlabeled data. These models need special design and hence are difficult to train especially for domain tasks, such as Cryo-Electron Tomography (cryo-ET) classification in computational biology. To address this challenge, we propose a novel method using knowledge transfer to boost uncertainty estimation in AL. Specifically, we exploit the teacher-student mode where the teacher is the task model in AL and the student is an auxiliary model that learns from the teacher. We train the two models simultaneously in each AL cycle and adopt a certain distance between the model outputs to measure uncertainty for unlabeled data. The student model is task-agnostic and does not rely on special training fashions (e.g. adversarial), making our method suitable for various tasks. More importantly, we demonstrate that data uncertainty is not tied to concrete value of task loss but closely related to the upper-bound of task loss. We conduct extensive experiments to validate the proposed method on classical computer vision tasks and cryo-ET challenges. The results demonstrate its efficacy and efficiency.
BibTeX
@inproceedings{wang2025boosting,
  author = {Tianyang Wang and Xi Xiao and Gaofei Chen and Xiaoying Liao and Guo Cheng and Yingrui Ji},
  title = {{Boosting Active Learning with Knowledge Transfer}},
  booktitle = {ICTAI},
  year = {2025},
  doi = {10.1109/ictai66417.2025.00164},
}
Figure