Causal foundation models are networks pre-trained on millions of simulated cause-and-effect systems. Given a new observational dataset as a prompt, they output a treatment effect or a causal graph in one forward pass, with no per-dataset fitting. They are fast and often more accurate than classical estimators, but they are trained on prompts of a few hundred to a few thousand rows. Real cohorts are often ten or a hundred times larger. Tabular models of this kind are known to degrade when the test-time sample count exceeds the training range, and the obvious workarounds are to throw data away, pay quadratic cost to retrain, or switch to an architecture that is sensitive to row order. This note argues that the failure has a specific, fixable cause, and that the fix can be derived from what a causal estimator ought to do as data grows, rather than learned.
The core idea fits in two sentences. Attention across the rows of a dataset has no positional encoding, so the only thing that changes as the dataset grows is the softmax denominator, which flattens attention weights as more rows compete. For the attention heads that act as covariate-balancing or local-smoothing operators, statistics already says how sharp those weights should become as the sample grows, so the per-head temperature can be set analytically from the training-time behavior, with no long-context data at all.
Core Claim
The known failure is documented for tabular in-context learners. GAMformer is a bi-attention network that predicts additive-model shape functions from synthetic data, and it degrades when inference sample sizes exceed its roughly 500–1,000-point training context. In language modeling, Length Generalization of Causal Transformers without Position Encoding showed that models without position encodings still fail at long lengths. The cause there is an "attention entropy spike": the softmax spreads its mass over too many tokens. The paper fixed this with HeadScale, a learned temperature per head (704 parameters), which kept PG19 perplexity at 18.3 at 16k tokens, while the unmodified model exceeded 1,000. Row-wise attention in causal foundation models is permutation-equivariant by design. It therefore removes the other known extrapolation failure, out-of-distribution rotary phase angles (Resonance RoPE), and isolates entropy dilution as the only remaining mechanism. That makes the sample-size setting an unusually clean test case for the entropy account.
What is new is the source of the correction. Towards Causal Foundation Model: on Duality between Optimal Balancing and Attention (CInA) proved that row-wise self-attention solves the dual of kernel optimal covariate balancing, so attention weights are balancing weights. Seen this way, a head with logit scale $\beta$ is a kernel smoother whose bandwidth is set by $\beta$. Classical nonparametric theory says the bandwidth should shrink with sample size at a known rate, not stay fixed. We distinguish two head types, which we identify from training-range behavior. The first type is selection heads, which retrieve a few specific rows. For these we keep attention entropy constant, in the spirit of HeadScale. The second type is smoothing or balancing heads, which average over a neighborhood. For these we shrink the effective bandwidth at the mean-squared-error-optimal kernel rate. Writing $N_0$ for the training context size, $N$ for the test sample size, and $d_h$ for the effective dimension of the head's key space, the two rules are
$$ \beta_h(N) = \beta_h(N_0)\,\frac{\log N}{\log N_0} \quad \text{(selection heads)}, \qquad \beta_h(N) = \beta_h(N_0)\left(\frac{N}{N_0}\right)^{2/(d_h+4)} \quad \text{(smoothing heads)}. $$
The first rule holds the softmax's effective number of attended rows constant. The second follows because the optimal kernel bandwidth scales as $N^{-1/(d+4)}$ for second-order smoothers, and $\beta$ acts like an inverse squared bandwidth for dot-product kernels on normalized keys. Both are hypotheses. The point of the project is to test which head obeys which rule, and whether the analytic rule matches or beats a learned temperature.
Research Program
We first build the diagnostic. We will take three amortized causal models that differ in output type. The first is a CInA-style balancing model, which estimates average treatment effects (ATE). The second is a CausalFM-style prior-data fitted network (Foundation Models for Causal Inference via Prior-Data Fitted Networks), which estimates conditional effects. The third is a TabPFN-CFM-style model (A Causal Foundation Model for Structure and Outcome Prediction), which predicts structure and interventional outcomes. Where released weights are not available, we will retrain them on their published synthetic priors at a fixed context size $N_0$. We will then sweep the test sample size from $N_0$ to the largest size exact attention fits in memory. State-Space Models for Tabular Prior-Data Fitted Networks reports TabPFN's memory ceiling at $2^{16}$ rows on an 80GB GPU, so chunked exact attention will be needed near that limit. Along the sweep we record per-head attention entropy alongside estimation error. The entropy account predicts that error growth should track entropy growth head by head, and that it should appear even though rows carry no positions.
Second, we classify heads using only training-range data. A head whose attention entropy is roughly flat in $N$ below $N_0$ is a selection head. A head whose entropy grows like $\log N$ is a diffuse smoother. For smoothers, we estimate $d_h$ from the intrinsic dimension of its key vectors. The rescaled model is then evaluated zero-shot at large $N$.
Third, we test the claim through the causal estimand rather than through perplexity-like proxies. For the balancing model, the dual view lets us compare the network's attention-derived weights at large $N$ against the explicit kernel-balancing solution on the same data. If the analytic rule is right, the two should converge as $N$ grows, while unscaled attention weights should drift toward uniform.
The multimodal reasoning gap and positional bias enter as constraints that shape the design, not as goals. Positional bias motivates staying with permutation-invariant attention. Choosing 'Right' from Wrong found input-order bias in every one of eight multimodal models, with p < 10^-4, and larger models were no better. The sequential state-space alternative needs repeated context permutations to suppress row-order effects. Any scaling fix must therefore preserve exact invariance, and we test this directly. The multimodal gap explains why we do not take the tempting route of reading cohorts off charts. Plot Twist reports 4.2% accuracy for ChartLlama on scatter-plot value extraction. MATH-Vision attributes 42.2% of GPT-4V errors to reasoning after correct perception. Both suggest that multimodal models fail precisely at aggregating many correctly perceived items. We include one small probe, described below, to check whether this aggregation failure grows with point count in the way the entropy account would predict. We do not claim a multimodal causal model.
Baselines
The naive reference point is each model applied unchanged at large $N$. This is the number every fix must improve on, and it shows how bad the problem actually is for causal estimands as opposed to classification accuracy.
The first strongest practical competitor is subsample-and-average. It draws repeated random subsets of size $N_0$, runs the model on each, and averages the estimates. It needs no modification, is permutation-invariant in expectation, and costs linear time in the number of subsets. We will give it the same wall-clock budget as our method. Our method beats it if, at $N \geq 8N_0$, error keeps decreasing with $N$, while subsample-averaging plateaus at the bias floor of an $N_0$-row estimator.
The second strongest competitor is learned HeadScale. We fine-tune per-head temperatures on a small slice of long-$N$ synthetic prompts, following Length Generalization of Causal Transformers without Position Encoding and matching its 0.03%-of-data budget. Our analytic rule wins if it matches learned HeadScale without any long-$N$ data. It also wins if it beats HeadScale when the test $N$ exceeds the lengths HeadScale was tuned on. We also run UniformScale, a single $\log N$ factor for all heads, which that paper found suboptimal. It isolates whether head typing matters.
The third competitor replaces attention with a bidirectional state-space model plus repeated context permutations (State-Space Models for Tabular Prior-Data Fitted Networks). It is retrained on the same causal prior with matched GPU hours. It wins on memory, so the comparison is error and row-order variance at fixed $N$.
For classical references, we use double machine learning and doubly robust estimators tuned by cross-validation on each dataset. On ACIC 2018 with 10k units, CInA reports ATE MAE 1.460, against 8.57 for double machine learning and 69.39 for RieszNet (Towards Causal Foundation Model). We treat 1.460 as the published bar our rescaled CInA-style model must match or improve at larger subsets of that benchmark. On Jobs, CausalFM reports PEHE 0.515, against 0.557 for CausalPFN and 0.563 for the X-learner. We have no reported number on Lacuna for any of these models as a function of sample size, and producing those curves is part of the contribution.
The key ablation swaps head assignments: smoothing rules applied to selection heads and vice versa, plus random assignment. If random assignment performs as well, the statistical story is wrong and only "some rescaling" matters.
Why not just…?
Why not just subsample to the training size and average? This is the most serious threat. For roughly linear estimands such as an ATE from balancing weights, averaging many subsamples may already recover most of the large-$N$ precision. If subsample-averaging matches our method on ACIC 2018 and the synthetic sweeps, the honest conclusion is that extrapolation is unnecessary for ATE. We would then narrow the claim to estimands where subsampling is known to lose: conditional effects at sparse covariate regions, and structure discovery, where rare dependencies only become detectable with more rows. The sweep reports both regimes separately so this outcome can be read off directly.
Why not just pretrain at a larger context? Pretraining cost is quadratic in rows, and GAMformer already reports 600 A100 hours at its modest context. More importantly, any finite pretraining context only moves the cliff further out. We include one model pretrained at $4N_0$ as an upper reference. If our rescaled $N_0$ model matches it at $4N_0$ and keeps improving beyond it, the cheaper route is justified.
Why not just use double machine learning once datasets are large? Classical estimators become strong precisely when $N$ is large. At 10k units on ACIC 2018, the published gap (8.57 versus 1.460) favors the amortized model, but that is one benchmark. If double machine learning overtakes every amortized variant beyond some $N$, we will report that crossover. It would bound where amortization is worth using, which is a useful result in itself.
Why not just use a state-space model to avoid attention entirely? It scales further in memory. However, it gives up exact row-permutation invariance and needs repeated permutations to compensate. Our row-shuffle variance measurement shows whether that cost is material for causal estimands.
Proposed Experiments
The synthetic sweep uses Erdős–Rényi structural causal models drawn from each model's own prior and from the random-Fourier out-of-distribution prior of TabPFN-CFM. Test sizes run from $N_0$ to $2^{16}$. We report ATE absolute error, conditional effect error (PEHE), interventional MSE (published bar 0.765, against 0.885 for the doubly robust learner), and adjacency AUROC (published bar 0.889). Each is shown as a curve in $N$, together with per-head entropy. We will also include linear Gaussian systems, where Demystifying amortized causal discovery with transformers shows structure is unidentifiable. There, more rows should not help direction recovery under any method. This is a sanity check that rescaling does not manufacture false confidence.
The semi-real evaluation uses ACIC 2018 subsets of increasing size and the Jobs data. Order sensitivity is measured as the variance of the estimate across 20 row shuffles and 20 column orders. The column-order test also covers order-scoring models such as TABORDER, whose column-presentation sensitivity has not been reported.
The small multimodal probe renders treatment-outcome scatter plots with 10 to 1,000 points from known models. It asks an open multimodal model for the sign of the effect and compares the result with the amortized model run on the true points. This tests whether multimodal accuracy falls with point count even when the visual pattern gets clearer.
Two decisive checks frame the project. The entropy account and the analytic rule both fail if per-head entropy does not predict error growth, or if randomly assigned rules match typed ones. The method is not worth its complexity if subsample-averaging matches it on every estimand.
References
- GAMformer: In-Context Learning for Generalized Additive Models
- Length Generalization of Causal Transformers without Position Encoding
- Resonance RoPE: Improving Context Length Generalization of Large Language Models
- Towards Causal Foundation Model: on Duality between Optimal Balancing and Attention
- Foundation Models for Causal Inference via Prior-Data Fitted Networks
- A Causal Foundation Model for Structure and Outcome Prediction
- State-Space Models for Tabular Prior-Data Fitted Networks
- Choosing 'Right' from Wrong: A Closer Look at Selection Bias in Spatial Multiple-Choice Questions in Large Multimodal Models
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Demystifying amortized causal discovery with transformers
- Inducing Causal Order through Tabular In-Context Learning