When a model trained on cardiologist-labelled ECGs persistently disagrees with the label, standard practice treats the disagreement as label noise and down-weights or relabels the example. That disagreement can actually mean three different things. The reader may simply have erred. The reader may have followed a local labelling habit that other hospitals do not share. Or the tracing may carry a real but sub-visual disease signature, one the cardiologist correctly called "normal" by visual criteria but that predicts a heart-failure admission or atrial fibrillation years later. Today's noise filters discard all three. They therefore discard exactly the early signals that the ECG-based screening literature is trying to find.
We propose to separate these cases using two pieces of evidence that a single hospital's training run cannot supply. The first is other hospitals: would models trained only on other sites' readers make the same disagreement? The second is time: does the disagreement predict events that are recorded later and diagnosed by means other than the ECG read? Disagreements that replicate across sites but have no outcome support are relabelled as local labelling habits. Disagreements that do not replicate across sites but are backed by outcomes are kept as subclinical signal. Disagreements supported by neither are treated as noise. The deliverable is a triage procedure and a training recipe built on it. Its test is whether cases it marks "subclinical" in one hospital carry measurably higher future event rates in held-out patients than cases it marks "noise."
Core Claim
Loss-based and entropy-based noise detectors cannot tell real subclinical pathology apart from reader error, because both look the same inside one site: high training loss on a confidently assigned label. Two signals can tell them apart. Cross-institutional replication separates site-idiosyncratic labelling from labelling that every site shares. A delayed outcome anchor separates disagreements that matter biologically from those that do not. Combining the two should keep subclinical cases that current filters throw away, without giving up the accuracy that noise cleaning buys on diagnostic labels at unseen hospitals.
Where This Sits
The closest work is Noise-Aware Generalization. It shows that methods for robustness to new domains and methods for learning with noisy labels undermine each other when both problems occur together. Its fix, DL4ND, fits a Gaussian mixture to per-sample losses, builds class prototypes for each domain, and relabels high-loss samples toward the nearest prototype in other domains. We reuse its central move of consulting other domains. We differ in two ways. DL4ND assumes every cross-domain mismatch is noise to be corrected toward the other domains' class, which is exactly wrong for subclinical cases. It is also formulated for single-label, mutually exclusive classes, whereas ECG statements are multi-label.
Two in-domain detectors constrain the design. LiNC learns a per-sample trust weight and fits a three-component mixture that already separates an "ambiguous" group from clean and corrupted samples. The Signed Entropy Integral tracks how prediction entropy evolves over training. Both papers concede two limits. First, a systematic clinician bias is memorized early and escapes detection. Second, their formulations assume hard single labels. Our cross-site test targets the first limit directly, because a bias that is systematic within one site need not be shared across sites.
Several papers motivate the outcome anchor. AnyECG predicts hundreds of future non-cardiac diagnoses from ECGs, but it is trained on billing codes with no noise modelling and no per-instance audit. Liu et al. report that systematic "misclassifications" of mild cognitive impairment as Alzheimer's foreshadow later decline, which is direct evidence that some errors are early signal. Xue et al. find severe coronary stenosis in ECGs that clinicians read as normal. Wilm et al. supply a validation tool: mitotic-figure slides annotated independently by 12 pathologists, where per-annotator disagreement is directly observable.
Research Program
We first train a multi-label ECG classifier on cardiologist statements, pooled across label-bearing sites: PTB-XL, Chapman-Shaoxing, CODE-15, and MIMIC-IV-ECG. Each site is kept identifiable. For every training example and every label, we compute an in-site disagreement score using LiNC's trust weight. This score defines the pool of suspect labels.
For each suspect label we then compute two further scores, both from models that never saw that example's site. The cross-site replication score is the probability, assigned by leave-one-site-out classifiers trained on the remaining sites' clinician labels, that the label should be flipped. Because sites differ in prevalence, a shift INPUTADAPTER and the domain-shift literature flag as entangled with hardware effects, these probabilities are recalibrated to each target site's label prevalence before comparison.
The outcome support score comes from a separate model trained on outcome-linked sites to predict future events. The primary source is MIMIC-IV-ECG linked to later admissions. UK Biobank follow-up, as used by AnyECG, is a second source where accessible. Outcomes are restricted to diagnoses that do not depend on reading this ECG, such as heart failure, valvular disease, or death, and to events beyond a fixed lag window. The score is the outcome model's predicted risk relative to patients with the same clinician label.
Triage then sorts each suspect label by these two scores into one of four groups. If both scores are high, the case is ambiguous and keeps a soft label. If replication is high and outcome support is low, it is a local habit and is relabelled toward the cross-site consensus. If replication is low and outcome support is high, it is subclinical: the clinician label is kept for the diagnostic head, and the example is up-weighted for a jointly trained risk head. If both are low, it is noise and is down-weighted. Thresholds are set on a development site, never on a test site. All decisions are made separately per label, which extends DL4ND to the multi-label case without nearest-prototype matching.
The final model has a diagnostic head and a risk head on a shared encoder. A second deliverable is the triage itself. Each flagged case carries an auditable reason, which answers the instance-level trust gap that AnyECG leaves open.
One known confound must be handled explicitly. Billing-code outcomes are themselves noisy, and some are partly triggered by ECG findings. The lag window and the exclusion of ECG-defined endpoints are meant to break this link. A sensitivity analysis will vary the lag, to show that outcome support is not just the clinician's read echoed through the record.
Baselines
The naive reference is plain empirical risk minimization on the pooled clinician labels with no cleaning, evaluated leave-one-site-out. For calibration of scale, a CNN ensemble on PTB-XL diagnostic statements reaches macro AUC of about 0.939 in the PTB-XL benchmark paper. That is an in-site, single-dataset number, so our cross-site figures are not expected to match it directly.
The strongest in-domain cleaners are LiNC and SEI. LiNC reports 80.23% last-epoch accuracy at 50% synthetic noise on MedMNISTv2 versus 58.82% for standard training, and noise-detection AUC of 0.9837 on OrganSMNIST. SEI reports more than 8% F1 improvement in error detection over Cleanlab, AUM and CORES on ISIC 2018 at 50% noise. Both will be run on the identical ECG encoder with per-label binary formulations.
The strongest method combining noise and domain shift is DL4ND, which can use our site IDs as its domain labels. It reports label recovery rising from 75.7% to 98.1% on RotatedMNIST with 30% noise, and a +12.5% gain on OfficeHome with 60% noise. No reported result was found on Lacuna for DL4ND, LiNC, or SEI on 12-lead ECGs.
An outcome-only model trained solely on future events tests whether the clinician labels add anything at all. Every method gets the same backbone, the same data splits, the same number of training epochs, and an equal hyperparameter search budget tuned on the same development site.
Three ablations isolate the key ingredient. Dropping the outcome score turns the method into a DL4ND-like cross-site relabeller. Dropping the replication score turns it into in-site outcome weighting. Replacing real hospitals with random pseudo-sites tests whether true institutional structure is what matters.
Beating the baselines means two things at once. First, matching or exceeding DL4ND and LiNC on held-out-site diagnostic macro AUC. Second, assigning a "subclinical" group whose future event rate in held-out patients is significantly higher than both the "noise" group and LiNC's "ambiguous" component, measured by hazard ratio with confidence intervals.
Why not just…?
Why not just scale the data, as AnyECG does? Scale averages away random errors but not systematic ones. LiNC and SEI both document that networks memorize systematic clinician habits early. In mitotic-figure assessment, Wilm et al. find that going beyond three annotators adds only about +0.013 F1. Scale also produces no per-case explanation. We will nonetheless initialize from the largest pretrained ECG encoder available to us, so that any gain is measured on top of scale rather than instead of it.
Why not just train on outcomes and drop clinician labels? This is the real threat. If outcomes were dense and available everywhere, the triage would be unnecessary. They are not. Follow-up exists at only some of our sites, and long-horizon events are rare and imbalanced, a point Holmström et al. make for sudden cardiac death. The outcome-only baseline settles the question. If it matches our model on both diagnostic AUC at label-only sites and outcome AUROC, the clinician-label pathway is not earning its place, and we would report that.
Why not just use LiNC's ambiguous mixture component as the "subclinical" bucket? LiNC may already capture part of the effect. The direct test is whether its ambiguous component is enriched for future events to the same degree as our subclinical group. Its own stated weakness is systematic biases that are memorized early, and that is precisely the habit-driven group our cross-site score is designed to pull apart.
Why not just run DL4ND? It would relabel subclinical cases toward other sites' "abnormal" prototypes or confirm them as "normal," and either way the outcome information is lost. The ablation without the outcome score reproduces this behaviour, so the difference in event-rate enrichment measures the cost directly.
Decisive Checks
Three checks decide the proposal. On the 12-pathologist mitotic dataset, we treat individual pathologists as "sites" and test whether the replication score recovers annotator-specific disagreements against the 3-consensus reference. This validates the habit-detection arm where ground truth exists. On ECGs, the main test is event-rate enrichment of the subclinical group in held-out MIMIC-IV-ECG patients, alongside diagnostic macro AUC on PTB-XL and Chapman-Shaoxing when each is held out entirely. The idea fails if the subclinical group shows no more enrichment than LiNC's ambiguous component, or if it disappears under the pseudo-site ablation.
References
- Noise-Aware Generalization: Robustness to In-Domain Noise and Out-of-Domain Generalization
- LiNC: Lightweight Noise Correction via Adaptive Label Refinement
- On Revisiting Entropy for Identifying Mislabeled Images
- AnyECG: Evolved ECG Foundation Model for Holistic Health Profiling
- Generalizable deep learning model for early Alzheimer’s disease detection from structural MRIs
- Screening for severe coronary stenosis in patients with apparently normal electrocardiograms based on deep learning
- Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Artificial Intelligence in Ventricular Arrhythmias and Sudden Death