Abstract
The ubiquity and rate of collection of cardiac signals produce large, unlabelled datasets. Active learning (AL) can exploit such datasets by incorporating human annotators (oracles) to improve generalization performance. However, the over-reliance of existing algorithms on oracles continues to burden physicians. To minimize this burden, we propose SoCal, a consistency-based AL framework that dynamically determines whether to request a label from an oracle or to generate a pseudo-label instead. We show that our framework decreases the labelling burden while maintaining strong performance, even in the presence of a noisy oracle.
Introduction
In the era of digital health, wearable sensors generate a constant stream of cardiac signals, such as electrocardiograms (ECGs) and photoplethysmograms (PPGs). While this provides a wealth of data, training deep learning models to interpret these signals requires expert annotation. This creates a significant bottleneck: physicians are expensive, busy, and in short supply.
Active Learning (AL) is the standard approach to this "limited label" problem, where an algorithm selects the most informative unlabelled samples for a human "oracle" to label. However, traditional AL research often overlooks two critical real-world challenges: the high annotation burden on medical experts and the reality of imperfect oracles who may provide noisy labels due to fatigue or disagreement. The SoCal framework addresses these issues by optimizing when to involve a physician and how to identify the most valuable data points.
Methods
The SoCal framework introduces a three-pronged strategy to enhance the efficiency and robustness of the active learning pipeline.
Bayesian Active Learning by Consistency (BALC)
Standard AL methods, such as Bayesian Active Learning by Disagreement (BALD), typically rely on Monte Carlo (MC) Dropout to perturb model parameters and identify samples where the model is uncertain. BALC extends this by perturbing both the model parameters and the input data itself. By evaluating the consistency of the model's output under these dual perturbations, BALC identifies samples where the decision boundaries are brittle, marking them as highly informative.
Selective Oracle Questioning (SoCal)
This core innovation moves beyond the binary choice of labeling or not. SoCal employs a secondary "oracle selection network" that predicts the likelihood of the main model misclassifying a specific sample. If the risk of error is high, the system requests a label from the physician. If the risk is low, the model generates a "pseudo-label" for the sample, integrating it into the training set without human intervention. This "trust but verify" approach allows the system to become more autonomous as it matures.
Area Under the Tracked Acquisition Function (AUTAF)
Rather than taking a single snapshot of uncertainty, AUTAF tracks the evolution of a sample’s informativeness over multiple training epochs. This temporal tracking provides a more stable and diverse set of candidates for labeling, preventing the model from being swayed by transient fluctuations in uncertainty.
Experimental Setup and Datasets
The framework was rigorously evaluated across four publicly available cardiac signal datasets:
- PhysioNet 2015 (D1 & D2): Focused on arrhythmia detection using PPG and ECG signals.
- PhysioNet 2017 (D3): ECG recordings categorized into Normal, AFib, Other, and Noisy classes.
- Cardiology ECG (D4): A complex dataset involving 12 different classes of cardiac arrhythmias.
Key Results and Insights
Efficiency and Workload Reduction
BALC demonstrated significantly faster learning rates compared to state-of-the-art methods. In several trials, BALC reached performance thresholds in half the time required by standard BALD. Furthermore, SoCal achieved an Oracle Ask Rate (OAR) of approximately 0.68, effectively reducing the physician's workload by 32% with negligible impact on model accuracy.
Robustness to Oracle Noise
A significant finding was SoCal's resilience to noisy oracles. In scenarios where human experts provided incorrect labels, SoCal often outperformed traditional AL. By selectively choosing when to query the oracle, the system avoided integrating erroneous human data into the training set, relying instead on its own consistent pseudo-labels for clearer samples.
Performance Gains
On the D2 dataset, SoCal achieved a Test AUC of 70.7, markedly higher than baseline methods like S-response (58.4) and ε-greedy (60.9). Additionally, the inclusion of AUTAF improved AUC by up to 11% in low-data regimes, highlighting the value of temporal informativeness tracking.
Conclusion and Future Work
SoCal represents a major advancement in making active learning practical for clinical settings. By dynamically managing the oracle's burden and improving acquisition through consistency, it bridges the gap between massive unlabelled datasets and scarce expert resources.
Future research will focus on refining the proxy signals used for oracle selection, particularly when initial labelled data is biased. Additionally, the authors aim to model dynamic oracle behavior, such as physician fatigue, and expand the framework to handle multiple experts with varying levels of clinical experience.
Summary updated