ALiBi Pooling as a Parallel Scan for Attributable ECG Monitoring

Cardiac monitors record for hours, but most ECG classifiers are trained on 10-second strips. They have no principled way to run on a 10-hour recording. When they do flag a rare condition, they cannot say when in the recording the evidence occurred. This proposal builds one small architectural piece to address both problems: a pooling layer that reads a long sequence of local heartbeat features and turns them into class scores. Its central observation is that attention with linear biases (ALiBi) becomes an ordinary linear recurrence when the query is a fixed, learned vector. ALiBi is the trick of subtracting a fixed, distance-proportional penalty from attention scores so that models trained on short inputs keep working on long ones. Because it becomes a linear recurrence, it can be trained in parallel with a prefix scan and run in streaming mode with constant memory. Because it is a normalized weighted average, every prediction splits exactly into per-timestep contributions with a known time horizon per head.

The application regime is class-imbalanced cardiology. Rare conditions such as myocardial infarction and hypertrophy are where current models fail (xLSTM-ECG). In long recordings, rare evidence is also brief: a paroxysmal arrhythmia episode is a tiny fraction of the record. The bet is that fixed, geometrically spaced decay rates do three things better than learned ones. They keep evidence at every timescale available to rare classes. They let a model trained on short windows run on full recordings. They yield explanations that are the model's own computation rather than a post-hoc estimate. Each of these is testable, and each could fail.

Core Claim

The research file frames a gap: ALiBi's "softmax shield" supposedly cannot be inserted into parallel linear recurrences without breaking associativity. That is true for full query-dependent softmax attention, but false for the case that matters in classification, where the query is a learned per-class vector.

Let $z_t$ be the local feature vector at time $t$, produced by a short-receptive-field convolutional encoder covering roughly two to three beats. For class $c$ and head $h$, let the content score be $s_{c,h,t} = u_{c,h}^\top z_t$ and the value be $v_{h,t} = W_h z_t$. Let $m_h$ be a fixed geometric slope as in Train Short, Test Long. The pooled representation at time $T$ is then

$$ r_{c,h}(T) = \frac{\sum_{t \le T} e^{\,s_{c,h,t} - m_h (T-t)}\, v_{h,t}}{\sum_{t \le T} e^{\,s_{c,h,t} - m_h (T-t)}} = \sum_{t\le T} \alpha_{c,h,t}(T)\, v_{h,t}. $$

This is exactly causal ALiBi softmax attention from a single query position. With $\lambda_h = e^{-m_h}$, the numerator and denominator obey

$$ N_T = \lambda_h N_{T-1} + e^{s_{c,h,T}} v_{h,T}, \qquad D_T = \lambda_h D_{T-1} + e^{s_{c,h,T}}, $$

so $r = N/D$. This is the diagonal linear recurrence of the Linear Recurrent Unit, with constant decay plus a content gate. It is computable by associative scan in $O(\log L)$ parallel depth and in $O(1)$ memory per step when streaming. In practice it needs the usual running-max rescaling for numerical stability.

The class logit is a linear readout of the $r_{c,h}$. It therefore decomposes exactly into per-timestep, per-head contributions $\alpha_{c,h,t}\, w_{c,h}^\top v_{h,t}$. Each head has a stated horizon of about $1/m_h$ samples, so explanations come with both a when and a how far back.

What is new is not ALiBi, attention pooling, or linear scans individually. The contribution is the claim, tested on imbalanced clinical data, that this specific restricted form gives three things at once. The first is length extrapolation from 10-second training windows to full ambulatory records. The second is faithful temporal attribution, exact by construction rather than estimated. The third is better minority-class performance than the same architecture with learned decay rates.

Relation to Prior Work

Two papers supply the method. ALiBi (Train Short, Test Long) gives the fixed geometric slopes and the train-short/test-long protocol. It was established only for full softmax attention in language modeling, and its authors attribute the gains mostly to a locality preference rather than long-range reasoning. That caveat is why we restrict long-range mixing to a single pooling stage instead of claiming deep long-context reasoning. The LRU (Resurrecting Recurrent Neural Networks) supplies the diagonal-recurrence-plus-scan machinery. Our design differs in fixing the decays rather than learning them, and in adding normalized content gating.

Two closer neighbors use ALiBi outside softmax-attention language modeling. TITAN (Multimodal Whole Slide Foundation Model for Pathology) uses ALiBi to extrapolate from crops to gigapixel slides and reports gains on rare-cancer retrieval, but it keeps quadratic attention and does not study the attributions. DeltaStack (Push, Pop, Parallelize) uses ALiBi-style extrapolation inside chunkwise-parallel linear attention, but only on code, math, and formal languages, with no attribution or imbalance analysis. LittleBird (LittleBird) shows the symmetric $|i-j|$ variant for bidirectional encoders. We use that variant only for offline whole-record analysis.

Four papers motivate the problem. HAN-ECG shows that attention over wave, beat, and window levels yields explanations cardiologists accept (missing P-waves, irregular R-R intervals). However, it uses sequential bidirectional RNNs on fixed 5-second strips, so it neither parallelizes nor extrapolates. MedMamba gets strong subject-independent results with selective state-space models and interprets inter-lead structure through a learned graph, leaving its temporal recurrence opaque. xLSTM-ECG reports persistent failure on HYP and MI, which defines our minority-class target.

One paper constrains the design. PaX shows that GPT-2, Mamba, and MedGemma sepsis models exhibit primacy bias, and that LIME mistakes position for importance. Since PaX only diagnoses this post hoc, we treat it as an evaluation tool. Our recency-biased pooling is a structural intervention whose effect on that bias is measurable with PaX's position-swapping test.

Coherent concept-based explanations report an interpretability constraint raising accuracy from 85.33% to 96.00% on PH². That result gives some reason to expect a constraint need not cost accuracy, though it was obtained in images, not time series.

World Model Augmentation attacks imbalance by synthesizing latent embeddings. It is complementary to our approach rather than competing, and we include it only as an optional add-on for the imbalance study.

Research Program

We build the model in three parts: a local convolutional encoder whose receptive field is capped at about 2.5 seconds, a bank of $H$ ALiBi pooling heads with per-class queries, and a linear readout. The receptive-field cap is deliberate. It is what makes the attribution exact at the level of short, clinically readable segments. It is also the model's main expressive risk, since rhythm patterns spanning many beats must be assembled by the pooling stage rather than by deep mixing. All long-range interaction happens in the pooling stage.

The slopes follow ALiBi's geometric schedule. Because the input is a physiological signal with a known sampling rate, we choose the schedule to span horizons from about one beat to several minutes. This turns the slope schedule from a heuristic into a stated set of clinical timescales.

The analysis has three threads, each asking whether a supposed benefit is real.

The first thread is extrapolation. We train only on 10-second windows and evaluate on full-length recordings in streaming mode. We check whether calibration and per-class AUROC hold as length grows by one to three orders of magnitude.

The second thread is attribution fidelity. Exactness alone does not imply usefulness, because a weight $\alpha_{c,h,t}$ can be exact yet clinically meaningless. We therefore measure deletion and insertion AUC, following PaX's protocol. We also run a planted-artifact test: inject a short synthetic marker correlated with a minority label in training only, then check whether the attributions localize it.

The third thread is imbalance. We test the hypothesis that learned decay rates, trained under heavy label skew, migrate toward the horizons that serve majority classes. Fixed slopes would then preserve the horizons rare classes need. We measure this by tracking the learned decay spectrum and the per-class gradient share across horizons. Every model receives the same class-weighted loss, so rebalancing is held constant rather than being part of the claim.

Baselines

The naive reference point is the same convolutional encoder followed by mean pooling over time. Global average pooling is what most 1D ECG classifiers implicitly do. It extrapolates trivially, but it dilutes brief events in long records. Beating it on full-record minority-class AUROC is the minimum bar.

The strongest existing fixed-window methods are the PTB-XL deep models. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL reports a macro AUC of approximately 0.939 for diagnostic statements with ensembles. xLSTM-ECG reports 87.59% accuracy and 91.33% AUC on PTB-XL. These numbers come from differing label granularities and protocols. We will therefore retrain a 1D ResNet/Inception-style network under our exact folds, loss, and tuning budget, and cite the published figures only as context.

On 10-second PTB-XL, we expect to match rather than beat the strongest retrained CNN. Our success criterion there is macro AUROC within 0.01 of it, while improving HYP and MI recall at a fixed specificity. For AF on MIT-BIH, the reference is HAN-ECG: 98.81% accuracy, 99.08% sensitivity, and 99.86% AUC on 5-second strips. We aim to match it on strips and to exceed a strip-voting version of it on full-record episode detection.

The strongest competitor on the central question is the same pipeline with a learned-decay recurrence instead of fixed-slope ALiBi pooling. Concretely, this is an LRU-parameterized pooling layer, and separately a Mamba layer as in MedMamba, with temporal explanations computed by Integrated Gradients. A transformer with ALiBi softmax attention over patch tokens is a further competitor on short windows.

All of these share our encoder, data splits, class-weighted loss, number of hyperparameter trials, and GPU-hour budget.

Three ablations isolate the key ingredient. The first sets $m_h = 0$, giving content-gated pooling with no recency. The second removes the content gate, giving a plain multi-scale exponential moving average. The third keeps the slopes but makes them learnable. If fixed slopes do not beat learnable slopes on minority classes or on extrapolation, the ALiBi component is not earning its place.

Why not just…?

Why not just use Mamba or an LRU with learned decays and explain it with Integrated Gradients? This is the most serious alternative, because such models already parallelize and already perform well on physiological data (MedMamba). The counter-evidence is partly indirect. PaX shows that post-hoc explainers on Mamba confuse position with importance. Learned decays also have no guarantee of stable behavior beyond the training length. The proposal settles this head-to-head in three ways: the learnable-slope ablation, deletion/insertion AUC of exact weights versus Integrated Gradients, and the planted-artifact test. If learned decays match on accuracy, extrapolation, and fidelity, our contribution shrinks to a cheaper explanation, and we will report it as such.

Why not just slide a CNN over the long recording and aggregate window scores with a max or a vote? For AF detection this could genuinely work as well. A sliding window with soft-max aggregation is close to our model with $m_h = 0$ restricted to one window. The difference is that sliding-window aggregation has a single horizon and no streaming notion of "how recently." Our heads have explicit multi-scale horizons. The MIT-BIH full-record experiment compares the two directly on episode onset latency and false alarms per hour. If voting matches us, the multi-scale recency claim fails for rhythm tasks.

Why not just use a transformer with ALiBi on the 10-second strips? At 100 Hz with patching, quadratic attention on 10 seconds is cheap. On short strips, the efficiency argument is therefore weak and we do not make it. The case rests on two things. One is constant-memory streaming over hours. The other is exact attribution, which a multi-layer transformer's attention maps do not provide. We include this transformer as a baseline and expect it to be competitive on PTB-XL accuracy.

Why not just fix imbalance with oversampling, focal loss, or latent augmentation? These operate on the loss or the data, while our hypothesis concerns which timescales the architecture keeps available. The two are orthogonal. Every model receives class-weighted loss. We also run a crossed condition adding the latent augmentation of World Model Augmentation, which raised linear-probe macro-AUROC on MIMIC-IV-ECG from 0.687 to 0.743. If fixed slopes help only without rebalancing, the imbalance claim is weak.

Proposed Experiments

On PTB-XL, we use the patient-stratified folds and report macro AUROC on superclasses and diagnostic statements. We also report per-class recall at 90% specificity for HYP and MI, and cross-dataset transfer to the Georgia 12-lead set. xLSTM-ECG reports 95.50% accuracy and 95.44% AUC on Georgia, used as context only.

On the MIT-BIH Atrial Fibrillation Database, we train on 10-second windows from held-out-patient splits and evaluate streaming over complete multi-hour records. Metrics are episode sensitivity, false alarms per hour, and onset latency. On PhysioNet 2017, whose recordings vary from 30 to 60 seconds, we train on 10-second crops and evaluate at full length. No reported result was found on Lacuna for a numeric baseline on this set.

As a secondary probe of positional bias, we apply the pooling layer over hourly variables in the PhysioNet/CinC 2019 sepsis data and run PaX's position-swapping test. The question is whether recency-biased pooling removes the primacy effect without losing early-admission risk factors. The challenge reports winning utility scores of 0.33 to 0.36.

Three checks would decide the proposal. The first is whether full-record minority-class AUROC degrades less with length for fixed slopes than for learned decays. The second is whether exact weights beat Integrated Gradients on deletion AUC and localize the planted artifact. The third is whether the learned-decay spectrum measurably drifts away from the horizons that rare classes use. If all three come out negative, fixed linear biases offer nothing beyond convenience for clinical sequences, and that negative result is itself worth reporting.