Interval-Aware Attention Smoothing for Patient-Timeline Foundation Models

A patient's medical record is a long, irregular sequence of events: diagnoses, prescriptions, lab results and notes, spread across years. Foundation models for these records are language models whose "words" are clinical events. They learn by predicting the next event, and they are then adapted to downstream predictions with few labels. One failure is well documented. On the EHRSHOT benchmark, a 141M-parameter model of this kind beats simple count-based gradient-boosted trees when labels are scarce. On long-horizon tasks, such as predicting a diagnosis a year ahead, the count baselines catch up with it or beat it once more labels arrive (EHRSHOT). A plausible culprit is that the model's view of a patient changes erratically from one visit to the next. Its attention, meaning the weights that decide which past events matter for the current prediction, can jump between unrelated comorbidities even when only a few hours have passed and nothing clinically important has happened.

We propose to add one term to the pre-training loss. When two consecutive visits look back at the same shared history, their attention over that history should not change much if little time has passed. Larger changes are allowed as the gap grows. The penalty is also designed to tolerate occasional sharp breaks such as an acute admission. In two sentences: smoothness penalties on attention already help in driving and video, but they assume evenly spaced time steps, and clinical records are anything but even. We make the smoothness penalty scale with elapsed time and test whether this, rather than generic regularization, closes the long-horizon gap of patient-timeline foundation models.

Core Claim

Penalizing how fast attention over a patient's shared history changes between consecutive visits, with a penalty that weakens as the time between them grows, produces more stable patient representations. Our claim is that this stability is what long-horizon clinical prediction needs.

The prediction is specific. Gains should concentrate on EHRSHOT's long-horizon new-diagnosis tasks and on the higher-label regime where count models currently catch up. They should come with measurably lower representation drift between nearby visits. They should not be reproduced by an equally strong generic attention regularizer, by an evenly spaced smoothness penalty, or by time-decay attention alone.

Positioning Against Prior Work

The method is borrowed from work on smoothing attention over time. Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction penalizes the squared difference between attention at consecutive time steps (a total-variation penalty) in driving forecasts. It reports better sample efficiency on rare cases, keeping accuracy with 30% of minority-case data while baselines degraded (same work). In video, MARN uses a coherence loss across frames, and weakly supervised action recognition pairs smoothness with sparsity so that boundaries stay sharp. All of these operate on evenly sampled frames or steps. We keep their core idea, an attention-level penalty applied during training rather than output filtering. We change the assumption about time.

Two lines of work motivate the gap from the clinical side. TALE-EHR scales each attention score by a learned polynomial function of elapsed time. Its ablation found that time-aware attention mattered more than the language-model code embeddings. That weighting is still applied independently for each pair of events, so nothing constrains how attention at one visit relates to attention at the next. TALE-EHR is therefore our strongest architectural competitor, and its mechanism is something our penalty is added on top of, not something it replaces. MedM-PLM aligns codes and notes, but within single admissions and with no notion of time. The heart-failure survival study ran an LSTM over frozen BlueBERT report embeddings. It reports that sequential gains were small and calls for attention-based longitudinal modeling. That supports the view that recurrence over frozen features is not the answer.

One result constrains the design. BioBART found that sentence-permutation pre-training hurt biomedical performance, which suggests clinical sequences carry order information that objectives must respect. For our penalty, this means it must not wash out real order-dependent transitions. That is why the penalty is robust and decays with the time gap rather than quadratic and uniform.

The domain-pre-training literature supplies the premise that the backbone should be trained on domain data rather than adapted from a general model. The PEFT comparison for code, MonoCoder and APEX all find that domain pre-training matters more than how the model is fine-tuned. We take this as given. We do not re-litigate it.

Research Program

The backbone is a causal transformer over coded patient events, built in the CLMBR style and trained by next-event prediction. It includes TALE-EHR-style elapsed-time attention weighting, so the penalty is tested on top of the strongest known time mechanism.

The penalty works as follows. For a chosen subset of heads, let $a_i$ be the attention distribution of the query at visit $i$ over all events before visit $i$. Let $\tilde a_{i+1}$ be the attention of visit $i+1$ restricted to that same prefix and renormalized. The added loss is

$$ \mathcal{L}_{\text{smooth}} = \lambda \sum_i g_\phi(\Delta t_i)\,\rho\big(\mathrm{JS}(a_i,\tilde a_{i+1})\big), $$

where $\Delta t_i$ is the elapsed time between visits $i$ and $i+1$, and $\mathrm{JS}$ is the Jensen–Shannon divergence. $\rho$ is a Huber-type function that is quadratic for small changes and linear for large ones, so that genuine breaks are penalized only lightly. $g_\phi$ is a small, decreasing, learned function of $\Delta t$, bounded below at zero, so the penalty fades for long gaps. $\lambda$ sets the overall strength.

Restricting the comparison to the shared prefix is what makes the penalty well defined. Without the restriction, the new events at visit $i+1$ would make every comparison look like a change. To keep $g_\phi$ from collapsing to zero, we fix its value at $\Delta t = 0$ and learn only its decay shape. To let some heads specialize in abrupt events, we apply the penalty to half the heads in each layer and report the effect of this choice.

Pre-training uses MIMIC-IV, the corpus TALE-EHR used. Evaluation transfers to EHRSHOT's longitudinal Stanford cohort with $k$ from 1 to 128. This setup also tests whether stable attention helps a model pre-trained on short, episodic ICU records generalize to multi-year outpatient timelines. EHRSHOT names exactly this shift as a limitation of MIMIC.

Analysis has two parts. First, we measure representation drift: the cosine distance between patient embeddings at visits less than 7 days apart with no new diagnosis, compared against drift across known acute events. Second, we correlate per-patient drift with long-horizon prediction error.

Baselines

The do-nothing reference is the same backbone pre-trained without any smoothing term, using identical data, steps and tuning budget. Every claimed gain is measured against it first.

The count-based gradient-boosted model from EHRSHOT is the competitor the hypothesis is aimed at. It is the method that matches or beats foundation models on long-horizon tasks when $k > 64$. We run it on identical EHRSHOT splits and $k$-shot samples, with its own standard hyperparameter search. EHRSHOT's summary gives no exact per-task numbers in this file ("no reported result found on Lacuna for EHRSHOT long-horizon AUROC"), so we reproduce it ourselves. Beating it means a higher mean AUROC on the long-horizon new-diagnosis tasks at $k = 128$, holding across resampled few-shot draws. Under the same criterion we also report CLMBR-T-base where its weights are obtainable. Its larger private pre-training corpus makes it a reference point, not a matched comparison.

TALE-EHR is the strongest existing method on our pre-training data. It reports a 0.926 average AUROC on nine-disease progression prediction and 0.902 Acc@5 on next-visit codes on MIMIC-IV (TALE-EHR). We re-implement it as our backbone without the penalty and confirm that we land near those figures before adding anything. Beating it means a statistically significant gain on EHRSHOT long-horizon transfer while not falling below it on the MIMIC-IV tasks. For MIMIC-III readmission, the nearest multimodal reference is MedM-PLM at 74.70% AUC (MedM-PLM). We note that number but do not claim to target it, because it uses notes and our core model does not.

Three ablations isolate the key ingredient, and each gets the same tuning budget for its strength parameter. The first is an evenly spaced total-variation penalty in the style of the smooth-attention-prior work, with $g$ held constant. The second is a quadratic rather than robust $\rho$. The third is a generic attention regularizer: an entropy penalty plus attention dropout, tuned to the same reduction in validation perplexity. The proposal survives only if the interval-aware version beats all three on long-horizon transfer.

Why not just…?

Why not just use count features and boosted trees, since they already win at long horizons? This is the most serious threat, because EHRSHOT shows exactly that. Our response is empirical. If the smoothed model does not beat the count model at $k = 128$ on long-horizon tasks, the proposal fails on its central claim, and we will report it that way. One piece of evidence argues the outcome is open. The heart-failure study found that hand-built features scored near-random C-index (~0.50), while domain-pretrained report features reached 0.713 (survival study). Learned representations therefore carry signal that counts miss. The question is whether that signal can be kept stable over long horizons.

Why not just smooth the model's predictions over time after training? A moving average over outputs removes visible flicker. It cannot change what the representation learned, and the claimed benefit is sample efficiency during adaptation. The trajectory work credits its rare-case data efficiency to the internal penalty (smooth attention prior), and RiT improved verb mAP from 62.0% to 64.0% by aggregating internal features rather than labels. We test this directly by applying an exponential moving average to the unsmoothed model's per-visit risk scores. If it matches our method on the long-horizon tasks, internal smoothing is unnecessary.

Why not just rely on time-decay attention, which already encodes elapsed time? Time decay sets how much an old event counts at one visit. It does not stop the ranking of old events from reshuffling between two visits a day apart. Still, TALE-EHR's ablation shows time-aware attention is powerful, so it could already deliver most of the stability. Our backbone includes it. The drift measurement tells us whether it already keeps drift low. If it does, and the penalty adds nothing, that is a clean negative result.

Why not just add any regularizer, since smoothing might merely reduce overfitting? This is why the matched generic regularizer exists. If entropy plus dropout, tuned to the same perplexity, gives equal gains, the temporal structure of the penalty is not doing the work.

Proposed Experiments

The main experiment pre-trains five variants on MIMIC-IV with matched compute: plain backbone, interval-aware penalty, evenly spaced penalty, quadratic penalty, and generic regularizer. Each is then evaluated on all 15 EHRSHOT tasks at $k \in \{1, 4, 16, 64, 128\}$ with linear probes, using the benchmark's protocol and repeated few-shot draws. We also report in-domain MIMIC-IV disease-progression AUROC and next-visit Acc@5 to check that smoothing does not cost next-event accuracy.

Several checks decide the outcome. The first: does the interval-aware model beat both the count model and the plain backbone on long-horizon diagnosis tasks at $k = 128$, while the evenly spaced and generic variants do not? The second: is the measured drift between nearby, clinically quiet visits lower, while drift across acute events is preserved? This is the over-smoothing test the video literature flags as an open problem. The third: does per-patient drift predict long-horizon error in the plain model? If it does not, the mechanism we propose is not the reason foundation models lose at long horizons, regardless of any aggregate gain.

The last experiment is the learned decay function $g_\phi$, inspected on its own. A sensible shape, strong within days and weak over months, supports the interval-aware design. A flat curve means the evenly spaced penalty would have sufficed.