ALPS: Active Learning via Perturbations

Dani Kiyasseh, Tingting Zhu, David A. Clifton

Abstract

Clinical settings are often characterized by abundant unlabelled data and limited labelled data. This is typically driven by the high burden placed on oracles (e.g., physicians) to provide annotations. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. Whereas previous work addresses either one of these elements independently, we propose an AL framework that addresses both. For acquisition, we propose Bayesian Active Learning by Consistency (BALC), a sub-framework which perturbs both instances and network parameters and quantifies changes in the network output probability distribution. For annotation, we propose SoQal, a sub-framework that dynamically determines whether, for each acquired unlabelled instance, to request a label from an oracle or to pseudo-label it instead. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.

Introduction

In healthcare, a "data paradox" exists: wearable devices generate massive amounts of unlabelled cardiac data, but expert physicians have limited time for annotation. Active Learning (AL) aims to bridge this gap by selecting the most informative data points for labeling. However, standard AL methods often rely on a single "snapshot" of model uncertainty, which can be noisy and unreliable in clinical environments. Furthermore, most frameworks assume an always-available, perfectly accurate oracle, ignoring the reality of busy, potentially inconsistent human experts.

To address these challenges, we introduce ALPS (Active Learning via Perturbations), a novel framework designed for cardiac signals that optimizes both data selection and annotation. ALPS identifies informative data points by perturbing both inputs and model parameters, while a "selective questioning" mechanism reduces the labeling burden by intelligently choosing when to use model-generated pseudo-labels.

Methodology

The ALPS framework introduces three key innovations to improve the robustness and efficiency of active learning:

Bayesian Active Learning by Consistency (BALC)

Standard methods like Monte Carlo Dropout (MCD) only perturb model parameters to gauge uncertainty. BALC extends this by perturbing both the model parameters and the input data itself (Monte Carlo Perturbations, or MCP). It identifies informative samples by looking for those where the model's output is most inconsistent under these dual nudges.

The BALC framework: comparing Monte Carlo Dropout (model perturbation), Monte Carlo Perturbations (input perturbation), and the combined BALC approach.

Area Under the Tracked Acquisition Function (AUTAF)

Instead of calculating informativeness at a single point in time, AUTAF tracks this value across multiple training epochs. By calculating the area under this temporal curve, the system captures a richer set of model hypotheses, making data selection more robust to training noise.

Selective Oracle Questioning (SoQal)

SoQal acts as a gatekeeper for human experts. It uses an "oracle selection network" to predict if the main model is likely to misclassify a specific unlabelled sample. If the risk is high, it requests a label from the physician; otherwise, it utilizes a pseudo-label generated by the model itself.

Experimental Setup

The framework was evaluated on four major publicly available cardiac signal datasets:

  • PhysioNet 2015 PPG ($D_1$): Photoplethysmogram signals for detecting five types of arrhythmia.
  • PhysioNet 2015 ECG ($D_2$): Electrocardiogram signals for the same five arrhythmias.
  • PhysioNet 2017 ECG ($D_3$): Single-lead ECG recordings for AFib detection.
  • Cardiology ECG ($D_4$): Single-lead ECG data covering 12 different arrhythmia classes.

Discussion and Insights

The Power of Input Perturbations

We found that perturbing the signal itself (MCP) is often more effective than just perturbing the model (MCD). Input perturbations are more interpretable and can reveal decision boundary issues that parameter changes might miss. Consistency serves as a powerful proxy for information: a model that gives wildly different answers for a slightly noisy version of the same heartbeat indicates that it has not yet learned that feature effectively.

Comparison of MCD and MCP: input perturbations can reveal decision boundary issues that parameter changes might miss.

Robustness to Human Error

Surprisingly, SoQal can outperform systems that request a label for every sample if the oracle is "noisy" (prone to error). By selectively pseudo-labeling samples it is confident about, it avoids incorporating expert mistakes into the training set.

Key Results

  • Faster Learning: BALC reached high performance levels significantly faster than standard methods. On the $D_2$ dataset, it achieved an AUC of ~0.69 in half the training time required by the state-of-the-art BALD method.
  • Performance Gains: Incorporating temporal information via AUTAF improved the model's AUC by up to 11% in low-data regimes.
  • Efficiency: SoQal reduced the number of label requests to the oracle by approximately 32% while maintaining competitive performance.
  • Resilience: In tests with a highly noisy oracle (80% error rate), SoQal maintained an AUC of ~0.59, whereas baseline methods dropped to ~0.50 (random guess).

Limitations and Future Work

SoQal currently relies on a zero-one loss signal from the initial labeled set; if this set is extremely biased, the selection network may struggle to generalize. Future work will explore multi-oracle models to reflect clinical environments with varying expertise levels and model "oracle dynamics," such as physician fatigue, to further improve real-world deployment viability.

Summary updated

Abstract

Clinical settings are often characterized by abundant unlabelled data and limited labelled data. This is typically driven by the high burden placed on oracles (e.g., physicians) to provide annotations. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. Whereas previous work addresses either one of these elements independently, we propose an AL framework that addresses both. For acquisition, we propose Bayesian Active Learning by Consistency (BALC), a sub-framework which perturbs both instances and network parameters and quantifies changes in the network output probability distribution. For annotation, we propose SoQal, a sub-framework that dynamically determines whether, for each acquired unlabelled instance, to request a label from an oracle or to pseudo-label it instead. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.
BibTeX
@inproceedings{kiyasseh2022alps,
  author = {Dani Kiyasseh and Tingting Zhu and David A. Clifton},
  title = {{ALPS: Active Learning via Perturbations}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=2JPWIDHyMo},
}
Figure