SoQal: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals

Dani Kiyasseh, Tingting Zhu, David A. Clifton

Abstract

Clinical settings are often characterized by abundant unlabelled data and limited labelled data. This is typically driven by the high burden placed on oracles (e.g., physicians) to provide annotations. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. Whereas previous work addresses either one of these elements independently, we propose an AL framework that addresses both. For acquisition, we propose Bayesian Active Learning by Consistency (BALC), a sub-framework which perturbs both instances and network parameters and quantifies changes in the network output probability distribution. For annotation, we propose SoQal, a sub-framework that dynamically determines whether, for each acquired unlabelled instance, to request a label from an oracle or to pseudo-label it instead. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.

Introduction

In healthcare AI, particularly for cardiac monitoring, there is a vast amount of unlabeled data generated by wearable sensors. However, the expertise required to label these signals (e.g., ECGs) is scarce and expensive. Active Learning (AL) aims to solve this by selecting the most informative samples for expert review. Traditional AL methods face two primary challenges: they often focus solely on model uncertainty, potentially missing samples that are sensitive to data perturbations, and they assume the "oracle" (the expert) is always available and perfectly accurate.

Methodology

The proposed framework addresses these challenges through two main components: BALC for acquisition and SoQal for annotation.

Bayesian Active Learning by Consistency (BALC)

Standard acquisition functions like Bayesian Active Learning by Disagreement (BALD) typically use Monte Carlo Dropout (MCD) to estimate model uncertainty by perturbing network weights. BALC extends this by incorporating input data perturbations. By measuring the consistency of the model's output distribution under both model and data perturbations, BALC identifies samples near decision boundaries more effectively than weight-only methods.

Comparison of active learning acquisition strategies: Monte Carlo Dropout (model perturbation), Monte Carlo Perturbations (data perturbation), and the proposed BALC (dual perturbation).

Selective Oracle Questioning (SoQal)

To reduce the burden on experts, SoQal introduces an "Oracle Selection Network." This network predicts the likelihood of the main model misclassifying a sample. If the model is likely to fail, the system requests a label from the oracle. Otherwise, it applies a pseudo-label. This approach is monitored using the Hellinger distance to ensure the selection network's reliability.

Area Under the Tracked Acquisition Function (AUTAF)

The authors also introduce AUTAF, a metric that tracks a sample's informativeness over multiple training epochs. This temporal context helps distinguish between samples that are consistently difficult and those that are only temporarily confusing to the model.

Experimental Results

The framework was evaluated on four cardiac signal datasets (PhysioNet 2015, PhysioNet 2017, and Cardiology ECG).

  • Acquisition Performance: BALC consistently outperformed state-of-the-art methods. On the D2 dataset, it reached target performance metrics twice as fast as baselines.
  • Workload Reduction: SoQal achieved an Oracle Ask Rate (OAR) of approximately 0.68, reducing the labeling burden by 32% while maintaining competitive performance.
  • Noise Robustness: SoQal demonstrated significant resilience to oracle noise. In scenarios with 80% label noise, it maintained an AUC of ~0.59, whereas traditional strategies plummeted to ~0.50.

Conclusion

The combination of BALC and SoQal provides a robust framework for active learning in clinical settings. By intelligently selecting informative samples and selectively querying experts, the system significantly reduces the annotation burden while maintaining high diagnostic accuracy, even when faced with imperfect human labels. Future work may explore multi-oracle settings to better reflect real-world clinical environments.

Summary updated

Abstract

Clinical settings are often characterized by abundant unlabelled data and limited labelled data. This is typically driven by the high burden placed on oracles (e.g., physicians) to provide annotations. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. Whereas previous work addresses either one of these elements independently, we propose an AL framework that addresses both. For acquisition, we propose Bayesian Active Learning by Consistency (BALC), a sub-framework which perturbs both instances and network parameters and quantifies changes in the network output probability distribution. For annotation, we propose SoQal, a sub-framework that dynamically determines whether, for each acquired unlabelled instance, to request a label from an oracle or to pseudo-label it instead. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.
BibTeX
@inproceedings{kiyasseh2022soqal,
  author = {Dani Kiyasseh and Tingting Zhu and David A. Clifton},
  title = {{SoQal: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=2JPWIDHyMo},
}
Figure