A 12-lead electrocardiogram (ECG) records one beating heart from twelve viewing angles. Most of what a cardiologist reads is how those views agree or disagree. A small ST-segment shift in a few chest leads, or a P wave visible in one lead but flat in another, is diagnostic precisely because the other leads say something different. We propose a self-supervised pretraining task, meaning one that learns from unlabeled recordings, built directly on that structure. The network must predict every lead using only the other leads, for all twelve leads at once, in a single forward pass. The mechanism is diagonal-masked self-attention, an attention layer in which each position is forbidden from looking at itself. We change what the "diagonal" means. A lead is barred from its own signal and also from every lead that is an exact linear combination of it. Without that second rule the task would be trivial, because the six limb leads are fixed sums and differences of just two measured signals.
The bet is that this pretraining produces representations needing few labels. It also yields a second output nearly for free: a per-lead map of "what this lead should look like given the rest of the heart" set against what it actually shows. We expect that disagreement to be a compact, physically meaningful feature for findings confined to a few leads. We will test the claim on PTB-XL and MIMIC-IV-ECG at label fractions from 1% to 100%. The decisive test is a compute-matched comparison against the obvious simpler route, which is hiding whole leads at the input and reconstructing them.
Core Claim
Two published models sit on either side of this idea. LAMAE shows that modelling cross-lead interactions improves label efficiency on MIMIC-IV-ECG-Ext-ICD. However, its attention over lead tokens lets each lead attend to itself, so nothing forces a lead's representation to come from the others. TransApp uses diagonally masked self-attention during masked-reconstruction pretraining. It does so only along time, on single-channel smart-meter series, and the file reports no label-fraction curves for it.
Our claim is narrower than "diagonal masking helps ECGs." Exclusion from oneself only becomes a useful learning signal for multi-lead signals if two conditions hold. First, it is applied at the level of lead groups that are closed under linear dependence. Second, a lead's own content is kept out through every path in the network, not just the attention diagonal. Under those conditions the model gives better low-label performance than unmasked cross-lead attention. Its single-pass cross-lead prediction residuals then add information that the pooled embedding alone lacks.
Why the Naive Version Fails, and the Design That Follows
Two lessons from the diagonal-masking literature shape the design. The first is a constraint from T-TA, a text autoencoder that predicts every word from the other words in one pass. It shows that masking the attention diagonal does not stop a deep network from copying its input. The skip connections that add a layer's input to its output carry token $i$'s own content into later layers, where it can be recovered. T-TA's remedy, which it calls input isolation, works as follows. Keys and values stay fixed to the input embeddings. The query stream for position $i$ starts from a position embedding alone, and it reads content only through masked attention.
TransApp and the fine-tuning regulariser TLM, which randomly blocks tokens from attending to themselves or being attended to, both omit this step. We therefore expect their form of masking to leak once stacked deeply. We will measure that leak directly rather than assume it.
The second lesson is specific to the domain. Leads III, aVR, aVL and aVF are exact linear combinations of leads I and II. A one-lead diagonal mask would let the network rebuild lead III by subtracting lead I from lead II. That is a copy shortcut in disguise, and it would teach the encoder nothing about cardiac electrophysiology.
The architecture has two streams. A per-lead content encoder is a small convolutional front end followed by temporal attention within each lead, with no mixing across leads. It turns each lead $j$ into patch embeddings $E_j$, which play the role of T-TA's fixed keys and values. A query stream is initialised for each lead $i$ and time patch $t$ from learned lead-identity and time-position embeddings only. Through several layers it cross-attends to the $E_j$ under the mask
$$ M_{(i,t),(j,s)} = \begin{cases} -\infty & \text{if } g(j) = g(i) \\ 0 & \text{otherwise,} \end{cases} $$
The symbols are defined as follows. $(i,t)$ indexes the query for lead $i$ at time patch $t$, and $(j,s)$ indexes the key for lead $j$ at patch $s$. $g(\cdot)$ assigns each lead to a group. The default grouping puts the six limb leads in one group and gives each chest lead V1–V6 its own group.
Query streams never attend to each other, so stream $i$ can see only the content of other groups. This holds exactly, by construction, rather than approximately. The pretraining loss reconstructs lead $i$'s patches from stream $i$ in both the time and frequency domains. The frequency-domain term follows LEAST, which found that adding a Fourier-domain target pulls attention away from the dominant QRS complex and toward subtle P and T waves.
Under the default grouping, the frontal-plane leads must be inferred from the chest leads, and each chest lead from its neighbours plus the frontal plane. This is the classical question of projecting one cardiac dipole onto different axes. We will also test a coarser two-group mask (limb leads versus chest leads) and a finer mask with limb-group closure but each chest lead alone.
For downstream tasks, a classifier receives the pooled outputs of the query streams. It also receives the per-lead residual between each lead's actual and predicted signal, computed in the same single pass. Input-masking schemes would need one pass per held-out group to produce the same residual map.
Research Program
The work has three parts. The first is pretraining on unlabeled MIMIC-IV-ECG waveforms, keeping the content encoder compact in the few-million-parameter range. This follows the finding of Benchmarking ECG FMs that a 3.8M-parameter structured state-space model, ECG-CPC, beat models 25 times larger in 5 of 7 clinical task categories. That result implies inductive bias, not scale, is the lever worth testing.
The second part is leakage analysis. We replace lead $i$'s raw input with the corresponding lead from a different patient and measure how much stream $i$'s prediction changes. The full design should be exactly invariant. The variants with only a diagonal mask, or a mask without group closure, should not be, and we will report by how much.
The third part is label-efficiency evaluation at fixed label fractions, under both linear probing and full fine-tuning. We separate the contribution of the representation from that of the residual map by training classifiers on each alone and on both together.
Baselines
The naive reference point is a supervised network of the same backbone trained from random initialisation on each label fraction. It anchors every curve and shows whether pretraining helps at all at 1% labels.
The strongest existing method is contrastive predictive coding, a pretraining task that predicts future latent features of the signal. On full-label PTB-XL, Mehari & Strodthoff report a fine-tuned macro AUC of 0.9418, against 0.9318 for the best purely supervised model and 0.927 under linear evaluation. Its compact successor ECG-CPC leads the 26-task benchmark of Benchmarking ECG FMs, but the file gives no per-task figure for it. We found no reported result on Lacuna for PTB-XL at 1%, 5% or 10% labels. We will therefore rerun CPC on our label splits with the same pretraining corpus, the same number of optimiser steps, and the same hyperparameter search budget, which is a fixed number of trials per method. Beating it means a higher macro AUC at 1–10% labels that holds over at least five label-subset seeds, while matching it within noise at full labels.
The closest architectural competitor is LAMAE-style unmasked cross-lead attention. For this we use our own two-stream model with the diagonal mask removed and the query stream allowed to see its own lead, trained on the same objective. This isolates self-exclusion from everything else.
The key competitor is a whole-lead input-masking autoencoder. It hides randomly chosen linear-dependence groups at the input and reconstructs them, with the same encoder and the same group closure. We match compute in floating-point operations per lead target: it gets as many forward passes as needed to see as many lead-reconstruction targets per update as our model sees in one pass.
The ablations are as follows. The TransApp-style variant uses a token-level diagonal mask in a standard stack without input isolation. A second variant has input isolation but a plain one-lead diagonal, with no group closure. TLM-style stochastic self-masking serves as a regulariser on an ordinary encoder. A last variant drops the residual features at fine-tuning time. We also include a linear lead-reconstruction baseline, a least-squares map predicting each lead group from the others, as a fixed transform without learning. Its residuals feed the same classifier, which tells us whether a learned model adds anything over textbook lead geometry.
Why not just…?
Why not just hide whole leads at the input and reconstruct them? This is the real threat. It trains the same cross-lead prediction target without touching attention, and it may match our classification numbers. The input-masking autoencoder baseline exists to settle this. If it matches our model at equal compute on both representation and residual features, the honest conclusion is that the gain comes from group-closed cross-lead prediction, not from attention masking. We would report it that way. Our design keeps one structural advantage, which T-TA demonstrated for text: every lead keeps its full context and every lead is a target in one pass. That advantage matters in practice only if residual maps help downstream tasks, so the residual-only classifier is part of the test.
Why not just use attention dropout or TLM-style random self-masking? TLM reports that surgical self-masking beats DropHead, which drops whole attention heads, by 0.5 points on GLUE. However, TLM applies the mask randomly and without isolation, and as a fine-tuning regulariser rather than a pretraining target. The leakage test predicts that such masks leave a lead's own content recoverable. The TLM ablation shows whether that matters for label efficiency.
Why not just scale the model or use ECG-CPC as is? Benchmarking ECG FMs found that scale was not decisive, and that ECG-JEPA, not the largest model, led below 1,000 labelled samples. Scale alone is therefore unlikely to be the right comparison. ECG-CPC itself is a serious competitor and is our strongest baseline. If it wins at low labels, cross-lead self-exclusion is not worth its complexity.
Why not just use the known lead geometry with a fixed linear transform? Fixed least-squares lead reconstruction captures the dipole projection with no learning at all. If its residuals give the same downstream gain as ours, the learned query stream is unnecessary. The linear lead-reconstruction baseline tests exactly this.
Proposed Experiments
We pretrain on MIMIC-IV-ECG. We evaluate on PTB-XL superclass, subclass and full-statement tasks at 1, 5, 10, 30 and 100% labels, and on MIMIC-IV-ECG-Ext-ICD under the same fractions. CPSC2018 serves as an out-of-distribution transfer check. The primary metric is macro AUC, with AUROC per ICD level.
We run four decisive checks. The first is the label-efficiency curve against CPC and the unmasked cross-lead model. The second is the compute-matched comparison with the input-masking autoencoder. The third is the leakage-invariance test across the masking ablations. Here we predict that variants which leak lose most of the low-label gain, which would confirm T-TA's warning carries over to biosignals.
The fourth check uses statement classes whose diagnostic signal sits in a few leads, such as localized myocardial infarction and conduction-block statements. On these we expect the residual features to help most, and on rhythm-level statements least. If the gain is uniform across classes, or absent where we expect it to be largest, the claim that cross-lead disagreement is the useful signal fails, even if the headline AUC improves.
References
- Foundation Model for Cardiac Time Series via Masked Latent Attention
- ADF & TransApp: A Transformer-Based Framework for Appliance Detection Using Smart Meter Consumption Series
- Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning
- TLM: Token-Level Masking for Transformers
- Masked Autoencoders that Feel the Heart: Unveiling Simplicity Bias for ECG Analyses
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- Self-supervised representation learning from 12-lead ECG data