Responsible Active Learning via Human-in-the-loop Peer Study

Yu-Tong Cao, Jingya Wang, Baosheng Yu, Dacheng Tao

Introduction

Active Learning (AL) has emerged as a critical technique for training deep learning models efficiently. By strategically selecting the most informative data points for human annotation, AL significantly reduces the labeling burden compared to traditional supervised learning. However, a significant bottleneck in current AL workflows is data privacy. Most existing frameworks require users to upload their entire unlabelled datasets to a centralized cloud server so that a "task learner" can evaluate which samples are most beneficial to label. For sensitive applications involving personal photographs, medical records, or proprietary industrial data, this "upload-all" requirement presents an unacceptable privacy risk.

This paper introduces Peer Study Learning (PSL), a "responsible" active learning framework designed to protect user privacy by isolating unlabelled data on the client-side. By leveraging a novel teacher-student architecture, PSL ensures that only the data points selected for labeling ever leave the user's device.

The Privacy Challenge in Active Learning

In a standard cloud-based AL setup, the cloud service hosts the target model (the task learner). To decide which data to label next, the task learner needs to process the unlabelled pool. This necessitates transferring the entire pool to the cloud. Even if only 10% of the data is eventually labeled, 100% of it is exposed.

The authors propose a paradigm shift: move the sampling logic from the cloud to the local device. However, running a massive task-learning model on a mobile or edge device is computationally prohibitive. PSL solves this by introducing lightweight "peer students" that act as proxies for the cloud-based "teacher."

The PSL Framework

The PSL framework mimics a classroom environment to achieve efficient and private data sampling. It consists of two main components: a heavy Teacher model residing in the cloud and a group of lightweight Peer Students residing on the local end-user's device.

Overview of the Peer Study Learning (PSL) framework, highlighting the separation between the local user's protected data and the cloud-based task learner.

The learning process is divided into three distinct phases:

  1. In-Class Peer Study: The peer students are trained on the small set of already labelled data. They learn not only from the ground-truth labels but also from the Teacher’s predictions via knowledge distillation. This ensures the students' behavior aligns with the more powerful cloud model.
  2. Out-of-Class Peer Study: This phase takes place entirely on the client-side using the unlabelled (protected) data. The peer students collaborate and attempt to reach a consensus on these samples. Because they are lightweight and diverse, their collective disagreement serves as a powerful signal.
  3. Peer Study Feedback: This is the core sampling mechanism. The system identifies "challenging" data points by calculating the discrepancy (KL divergence) between the predictions of different peer students. Samples with the highest discrepancy—where the "students" disagree most—are deemed the most informative. Only these specific samples are sent to the human oracle for labeling and subsequently shared with the cloud-based Teacher.

Experimental Results

The researchers evaluated PSL across several standard benchmarks, including CIFAR-10, CIFAR-100, and Tiny ImageNet for image classification, as well as the Cityscapes dataset for semantic segmentation.

Performance comparison of PSL against state-of-the-art active learning methods across CIFAR-10, CIFAR-100, and Tiny ImageNet datasets.

Superior Accuracy and Robustness

As shown in the performance curves, PSL consistently achieves higher mean accuracy than established methods like VAAL, LL4AL, and BADGE. For instance, on CIFAR-10, PSL reached an accuracy of 78.06% with only 3,000 labels. Furthermore, the framework demonstrated remarkable robustness to "noisy" labels, maintaining high performance even when the human oracle provided incorrect annotations—a common scenario in real-world crowdsourcing.

Extreme Efficiency

One of the most striking advantages of PSL is its computational efficiency. While traditional AL methods often use sampling models nearly as large as the task learner itself, PSL utilizes extremely small student models (e.g., ResNet-8).

  • Model Complexity: The PSL sampling model uses only 0.16M parameters, compared to the 11M–23M parameters required by competing methods.
  • Sampling Speed: PSL is orders of magnitude faster, taking just 0.001 seconds to sample data, whereas methods like BADGE can take over 17 seconds for the same task.

Conclusion and Future Work

Peer Study Learning (PSL) represents a significant step toward responsible AI. By isolating unlabelled data and using a committee of lightweight students to identify informative samples, it provides a privacy-first alternative to traditional active learning without sacrificing performance.

While the paper demonstrates the potential for integrating PSL into Federated Learning (FL) environments, challenges remain regarding "non-IID" data distributions across different clients. Future research will likely focus on optimizing these decentralized interactions to ensure that the benefits of PSL can be scaled across diverse, real-world user populations while maintaining strict privacy guarantees.

Summary updated

Abstract

Active learning has been proposed to reduce data annotation efforts by only manually labelling representative data samples for training. Meanwhile, recent active learning applications have benefited a lot from cloud computing services with not only sufficient computational resources but also crowdsourcing frameworks that include many humans in the active learning loop. However, previous active learning methods that always require passing large-scale unlabelled data to cloud may potentially raise significant data privacy issues. To mitigate such a risk, we propose a responsible active learning method, namely Peer Study Learning (PSL), to simultaneously preserve data privacy and improve model stability. Specifically, we first introduce a human-in-the-loop teacher-student architecture to isolate unlabelled data from the task learner (teacher) on the cloud-side by maintaining an active learner (student) on the client-side. During training, the task learner instructs the light-weight active learner which then provides feedback on the active sampling criterion. To further enhance the active learner via large-scale unlabelled data, we introduce multiple peer students into the active learner which is trained by a novel learning paradigm, including the In-Class Peer Study on labelled data and the Out-of-Class Peer Study on unlabelled data. Lastly, we devise a discrepancy-based active sampling criterion, Peer Study Feedback, that exploits the variability of peer students to select the most informative data to improve model stability. Extensive experiments demonstrate the superiority of the proposed PSL over a wide range of active learning methods in both standard and sensitive protection settings.
BibTeX
@misc{cao2022responsible,
  author = {Yu-Tong Cao and Jingya Wang and Baosheng Yu and Dacheng Tao},
  title = {{Responsible Active Learning via Human-in-the-loop Peer Study}},
  year = {2022},
  eprint = {2211.13587},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.13587},
}
Figure