SoCal: Selective Oracle Questioning for Consistency-based Active Learning of Cardiac Signals

Dani Kiyasseh, Tingting Zhu, David A. Clifton

Abstract

The ubiquity and rate of collection of cardiac signals produce large, unlabelled datasets. Active learning (AL) can exploit such datasets by incorporating human annotators (oracles) to improve generalization performance. However, the over-reliance of existing algorithms on oracles continues to burden physicians. To minimize this burden, we propose SoCal, a consistency-based AL framework that dynamically determines whether to request a label from an oracle or to generate a pseudo-label instead. We show that our framework decreases the labelling burden while maintaining strong performance, even in the presence of a noisy oracle.

Introduction

In the era of digital health, wearable sensors generate a constant stream of cardiac signals, such as electrocardiograms (ECGs) and photoplethysmograms (PPGs). While this provides a wealth of data, training deep learning models to interpret these signals requires expert annotation. This creates a significant bottleneck: physicians are expensive, busy, and in short supply.

Active Learning (AL) is the standard approach to this "limited label" problem, where an algorithm selects the most informative unlabelled samples for a human "oracle" to label. However, traditional AL research often overlooks two critical real-world challenges: the high annotation burden on medical experts and the reality of imperfect oracles who may provide noisy labels due to fatigue or disagreement. The SoCal framework addresses these issues by optimizing when to involve a physician and how to identify the most valuable data points.

Methods

The SoCal framework introduces a three-pronged strategy to enhance the efficiency and robustness of the active learning pipeline.

Bayesian Active Learning by Consistency (BALC)

Standard AL methods, such as Bayesian Active Learning by Disagreement (BALD), typically rely on Monte Carlo (MC) Dropout to perturb model parameters and identify samples where the model is uncertain. BALC extends this by perturbing both the model parameters and the input data itself. By evaluating the consistency of the model's output under these dual perturbations, BALC identifies samples where the decision boundaries are brittle, marking them as highly informative.

Comparison of Active Learning acquisition strategies: Monte Carlo Dropout, Monte Carlo Perturbations, and the proposed Bayesian Active Learning by Consistency (BALC).

Selective Oracle Questioning (SoCal)

This core innovation moves beyond the binary choice of labeling or not. SoCal employs a secondary "oracle selection network" that predicts the likelihood of the main model misclassifying a specific sample. If the risk of error is high, the system requests a label from the physician. If the risk is low, the model generates a "pseudo-label" for the sample, integrating it into the training set without human intervention. This "trust but verify" approach allows the system to become more autonomous as it matures.

Area Under the Tracked Acquisition Function (AUTAF)

Rather than taking a single snapshot of uncertainty, AUTAF tracks the evolution of a sample’s informativeness over multiple training epochs. This temporal tracking provides a more stable and diverse set of candidates for labeling, preventing the model from being swayed by transient fluctuations in uncertainty.

Experimental Setup and Datasets

The framework was rigorously evaluated across four publicly available cardiac signal datasets:

  • PhysioNet 2015 (D1 & D2): Focused on arrhythmia detection using PPG and ECG signals.
  • PhysioNet 2017 (D3): ECG recordings categorized into Normal, AFib, Other, and Noisy classes.
  • Cardiology ECG (D4): A complex dataset involving 12 different classes of cardiac arrhythmias.

Key Results and Insights

Efficiency and Workload Reduction

BALC demonstrated significantly faster learning rates compared to state-of-the-art methods. In several trials, BALC reached performance thresholds in half the time required by standard BALD. Furthermore, SoCal achieved an Oracle Ask Rate (OAR) of approximately 0.68, effectively reducing the physician's workload by 32% with negligible impact on model accuracy.

Robustness to Oracle Noise

A significant finding was SoCal's resilience to noisy oracles. In scenarios where human experts provided incorrect labels, SoCal often outperformed traditional AL. By selectively choosing when to query the oracle, the system avoided integrating erroneous human data into the training set, relying instead on its own consistent pseudo-labels for clearer samples.

Performance Gains

On the D2 dataset, SoCal achieved a Test AUC of 70.7, markedly higher than baseline methods like S-response (58.4) and ε-greedy (60.9). Additionally, the inclusion of AUTAF improved AUC by up to 11% in low-data regimes, highlighting the value of temporal informativeness tracking.

Conclusion and Future Work

SoCal represents a major advancement in making active learning practical for clinical settings. By dynamically managing the oracle's burden and improving acquisition through consistency, it bridges the gap between massive unlabelled datasets and scarce expert resources.

Future research will focus on refining the proxy signals used for oracle selection, particularly when initial labelled data is biased. Additionally, the authors aim to model dynamic oracle behavior, such as physician fatigue, and expand the framework to handle multiple experts with varying levels of clinical experience.

Summary updated

Abstract

The ubiquity and rate of collection of cardiac signals produce large, unlabelled datasets. Active learning (AL) can exploit such datasets by incorporating human annotators (oracles) to improve generalization performance. However, the over-reliance of existing algorithms on oracles continues to burden physicians. To minimize this burden, we propose SoCal, a consistency-based AL framework that dynamically determines whether to request a label from an oracle or to generate a pseudo-label instead. We show that our framework decreases the labelling burden while maintaining strong performance, even in the presence of a noisy oracle.
BibTeX
@inproceedings{kiyasseh2022socal,
  author = {Dani Kiyasseh and Tingting Zhu and David A. Clifton},
  title = {{SoCal: Selective Oracle Questioning for Consistency-based Active Learning of Cardiac Signals}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=2JPWIDHyMo},
}
Figure