Some recent tabular models skip per-dataset training entirely. You hand them a table of labelled examples and they output a fitted model in one forward pass. This is called in-context learning. GAMformer does this for generalized additive models, which are interpretable models that sum one learned risk curve per feature. Its authors report that it holds up under class imbalance and label noise on ICU mortality in MIMIC-II, where it recovers clinically sensible risk curves. They also report that it breaks when the table holds more rows than it saw in pre-training, roughly 500 to 1,000. In clinical data this limit bites hard. With 3% mortality, a 1,000-row context holds about 30 deaths. Any risk curve in a region where deaths are concentrated, such as very low oxygenation ratios or extreme heart rates, is then estimated from a handful of patients.
The proposal is to fix this failure at inference time with almost no new parameters. We first test whether the failure is mainly the attention-dilution effect known from language models. When a model attends over many more items than it was trained on, its softmax weights spread out and lose sharpness. If that is the mechanism, the fix borrowed from HeadScale applies: a per-head attention temperature that grows with the logarithm of the number of rows, fitted only on synthetic tasks. We then ask the question that matters clinically. Does a single 20,000-row context give better risk curves for the minority class than the cheap alternative, which is to cut the data into balanced 1,000-row chunks and average the results? We predict it does, and only in a specific regime: regions of feature space where positives are sparse. We say below why averaging cannot close that gap.
Core Claim
Our hypothesis concerns the attention across rows (patients) in a tabular in-context learner. We expect that, unlike in a language model, this attention carries no positional encoding, because shuffling the patient rows should not change the fitted model. If so, length extrapolation failure here has nothing to do with rotary phase distortion (Resonance RoPE, Zhou et al.). It would then be purely the entropy spike that Wang et al. diagnosed in transformers without position encodings.
The file does not describe GAMformer's row encoding, so this is the first thing we check. If the hypothesis holds, the remedy is rescaling the attention temperature, not remapping positions. Methods built on remapping positions, such as SelfExtend and BiPE, have nothing to act on.
The claim specific to class imbalance is about bias, not variance. Think of a small-context in-context estimate as a smoothed estimate. In feature regions with few positives, it shrinks toward its learned prior. That shrinkage is a systematic error, and averaging many small-context estimates over different negative subsets cannot remove it. Every chunk sees the same scarce positives, and the scarcity comes from the chunk size, so each chunk shrinks in the same direction. A model that can actually use all rows sees more positives per region and shrinks less. The new result would therefore be a measured, region-resolved advantage of extrapolated full context over balanced chunk-ensembling, concentrated in sparse-positive regions. Equally publishable is a clean demonstration that no such advantage exists, which would tell practitioners that ensembling is enough.
Research Program
The first stage is diagnosis. We run a GAMformer-style model, using released weights if available or otherwise a smaller replica trained on the same kind of synthetic structural-causal-model prior, at context sizes from 500 to 32,000 rows. At each size we log the per-head entropy of the row attention. The prediction from the no-position-encoding analysis is that entropy rises roughly with the logarithm of the row count $N$, and that the degradation of the fitted risk curves tracks that rise. If degradation appears without an entropy spike, the mechanism is something else, such as the output layer expecting activations of a certain size. Stage two is then redesigned before any clinical experiments.
The second stage is the fix. Each row-attention head $h$ divides its logits by a scale that depends on context size:
$$s_h(N) = 1 + \alpha_h \log(N / N_0),$$
Here $N_0$ is the largest context size seen in pre-training, and $\alpha_h$ is one scalar per head. We fit the $\alpha_h$ on synthetic prior tasks drawn at sizes $N > N_0$, with the backbone frozen. The fitting objective is the error of the model's output against the known ground-truth shape functions. No clinical data touches the fitting, which keeps the clinical evaluation clean. The log form follows from the softmax denominator growing linearly in $N$, so the logits must grow like $\log N$ to keep the attention mass on relevant rows constant.
We also test a class-aware variant. It uses separate scales for attention onto positive-labelled and negative-labelled context rows. The motivation is that the number of positives grows much more slowly than the total row count. The shared-scale version is the main method; the class-aware version is a targeted hypothesis.
Row attention at 32,000 rows costs about $10^9$ attention entries per head per layer. That is feasible with memory-efficient kernels on one GPU. We do not need the retrieval machinery of DRT at this scale, and we note it as the route if scale grows by another order of magnitude.
Data and Evaluation Setting
The synthetic testbed uses tasks drawn from the structural-causal-model prior. Prevalence is controlled from 50% down to 1%, and we plant risk structure inside sparse-positive regions. Because the true shape functions are known there, we can measure curve error separately in dense and sparse regions.
The clinical tasks are in-hospital mortality from aggregated tabular features on four cohorts:
- MIMIC-II, to match GAMformer's own setting;
- MIMIC-III;
- MIMIC-IV, with 65,366 admissions;
- eICU, with 157,883 admissions, as a cross-site shift.
We add PhysioNet Challenge 2012 mortality as a small, sparse cohort. The OpenML suite used by GAMformer, with imbalance induced by subsampling positives, checks that results are not specific to clinical data.
The metrics are as follows:
- Discrimination: area under the precision-recall curve (AUPRC) is primary, following the warning in Revisiting Clinical Outcome Prediction for MIMIC-IV that AUROC hides minority failure. AUROC is secondary.
- Calibration: expected calibration error.
- Synthetic curves: shape-function error, reported separately in the dense and sparse regions.
- Clinical curves: the bootstrap stability of risk curves in the tails, such as the U-shaped heart-rate curve GAMformer recovers.
All of these are measured as a function of the number of training patients available. This learning curve is the real object of interest.
Baselines
The naive reference point is GAMformer run as published: a random subsample of $N_0$ rows, with no rescaling. A second naive point is the unmodified model fed the full table. That run should reproduce the reported collapse, and it confirms that the extrapolation problem exists in our setup.
The strongest competitor is balanced chunk-ensembling. All positives, capped at half of $N_0$, are combined with disjoint negative subsets to form many $N_0$-row contexts. We average the resulting shape functions and apply the analytic intercept correction for the true base rate, since a balanced sample shifts only the model's intercept. This baseline is cheap and needs no model changes. It exploits all the data in a variance-reducing way, which makes it the honest bar.
It gets the same frozen backbone and the same data. Its one tuning knob, the number of ensemble members, gets a search budget equal to what we spend fitting the $\alpha_h$. Its compute is matched to the full-context run in GPU-seconds.
Beating it means four things:
- a statistically reliable AUPRC gain on at least three of the four mortality cohorts;
- lower shape-function error in sparse-positive regions on synthetic tasks;
- no loss in dense regions;
- no loss in calibration.
The classical references are Explainable Boosting Machines, the GAMs fitted the conventional way by gradient boosting that GAMformer was compared against, and gradient-boosted trees. Both are tuned with the same hyperparameter budget on the full data. They require no extrapolation at all, so they bound how much in-context learning has to gain.
For an external anchor on MIMIC-IV mortality, ALFIA reports AUPRC 0.577 for an AutoGluon ensemble and 0.566 for FT-Transformer under its first-24-hour protocol. ALFIA itself reaches 0.585 using clinical notes. We will reproduce that cohort definition where possible and report against 0.577 as the tabular number to match. If the cohort cannot be matched exactly, we will say so rather than compare across protocols. We found no reported AUROC or AUPRC figure on Lacuna for GAMformer on MIMIC-II or MIMIC-III, so its numbers there must come from our own reruns.
Two ablations isolate the key ingredient. The first replaces the fitted $\alpha_h$ with one global constant, which tests whether per-head structure matters. The second sets $\alpha_h = 0$ but keeps everything else, which tests whether gains come from rescaling or merely from feeding more rows. The class-aware variant is compared against the shared-scale version as a third ablation. An upper-bound control fine-tunes the whole model on synthetic tasks at large $N$, which shows how much a cheap fix leaves on the table.
Why not just…?
Why not just subsample a balanced context and correct the intercept? For risk curves in regions where positives are reasonably dense, this probably works, and we expect it to tie our method there. Its weakness is structural. Each balanced context still holds at most a few hundred positives, so curves in sparse-positive regions are shrunk toward the prior. A single context cannot fix that. The synthetic sparse-region error, measured against known ground truth, settles whether this weakness matters in practice.
Why not just ensemble many small contexts? This is the real threat. It is the strongest baseline above because it uses all the data. Averaging lowers variance, but it leaves the shared shrinkage bias of small contexts untouched. Our prediction is therefore specific: the gap between our method and ensembling should grow as positives become sparser within a region, and vanish where they are dense. If the gap is flat or absent, we conclude ensembling suffices and report that as the finding. That would be a useful negative result for anyone deploying tabular in-context learners on hospital-scale cohorts.
Why not just use Explainable Boosting Machines or gradient-boosted trees on the full table? They have no context limit, and they may well win on raw AUPRC at large $N$. GAMformer's reported advantage is greater resilience to class imbalance and label noise than EBMs, along with amortized fitting. That advantage only matters if it survives at realistic cohort sizes, which is exactly what is untested. We include both classical models at matched tuning budgets so the answer is visible in the learning curves.
Why not just fine-tune the model at longer contexts? It would likely work best, which is why we run it as an upper bound. The original pre-training took 600 A100-hours. A fix that matches most of the fine-tuned gain while adding only one scalar per head is the more useful result. HeadScale's 704-parameter repair on PG-19 is the precedent: it took perplexity at 16,384 tokens from above 1,000 to 18.3.
Why not use SelfExtend or BiPE, the standard long-context fixes? Both work by remapping or decomposing token positions. If the diagnosis stage confirms that rows carry no positions, these methods have nothing to act on. That confirmation is the first thing we check.
Decisive Checks
Three outcomes would decide the proposal.
First, if row-attention entropy does not rise with $N$ in the frozen model, the mechanism is wrong. We stop and redesign before any clinical runs.
Second, if per-head rescaling does not restore synthetic shape-function error at 8 to 32 times $N_0$, at least to the level the model achieves at $N_0$, the cheap fix fails. Only the fine-tuned upper bound remains.
Third, the main empirical claim stands or falls on one comparison: the sparse-region advantage over intercept-corrected balanced chunk-ensembling. It must be positive on synthetic tasks and must carry over to AUPRC on MIMIC-IV and eICU at matched compute. If extrapolation wins on synthetic data but not clinical data, the implication is that real ICU risk does not live in sparse-positive regions of aggregated features, which is itself worth knowing.
References
- GAMformer: In-Context Learning for Generalized Additive Models
- Length Generalization of Causal Transformers without Position Encoding
- Resonance RoPE: Improving Context Length Generalization of Large Language Models
- On the token distance modeling ability of higher RoPE attention dimension
- LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning
- Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
- Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
- Revisiting Clinical Outcome Prediction for MIMIC-IV
- Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion