TL;DR
Tabular In-Context Learners (ICLs) like TabPFN can fail silently by relying on spurious signals hidden within composite feature representations (e.g., equipment artifacts bundled alongside real health data). This paper proves that this "spurious routing" is theoretically unavoidable and is actually worsened by larger context sizes or more expressive models. To fix this, the authors introduce "S-swap augmentation," a lightweight, training-free context manipulation that reduces spurious routing by up to 98.8%.
What Problem Does This Paper Address?
In-context learning (ICL) allows models to adapt to new tabular tasks at inference time using just a few labeled examples passed into the context window. However, this flexibility has a major blind spot.
If a single measured feature $X$ bundles a true causal signal $C$ (e.g., a patient's actual health status) with a spurious signal $S$ (e.g., a systematic artifact from a specific hospital's equipment), the model cannot inherently tell them apart. If $S$ correlates with the outcome $Y$ via an unobserved confounder $U$ (like patient demographics) within the training environment, the model will rationally but incorrectly route its predictions through $S$ instead of $C$.
When deployed in a new environment where the equipment artifact changes, the model fails silently. The authors formalize this failure mode as spurious routing in composite representations.
Methods
The authors introduce the Causal Sensitivity Ratio (CSR), a perturbation-based diagnostic metric that quantifies whether a model is more sensitive to causal or spurious variations. They mathematically analyze ridge ICL (a linear model baseline) and empirically test TabPFN (a state-of-the-art transformer-based tabular ICL).
To mitigate spurious routing without retraining the frozen models, they propose two lightweight, context-level interventions that only require weak environment labels:
- Environment-Stratified Context Construction: Sampling context examples evenly across different environments to break the localized $S–Y$ correlation.
- S-swap Augmentation: Swapping the spurious components ($S$) of context examples with those drawn from different environments while keeping the label ($Y$) fixed. This explicitly signals to the model in-context that $S$ does not reliably predict $Y$.
Datasets
- Synthetic SCM Data: A controlled structural causal model where dimensions of $X$ are explicitly partitioned into causal, spurious, and noise subspaces to validate theoretical predictions.
- scIB Human Pancreas Dataset: A real-world single-cell genomics dataset containing 16,382 pancreatic islet cells across 5 sequencing technologies (acting as environments) with real batch effects. A controlled confounding path was injected to test the methods under realistic, unknown-$S$ conditions.
Key Insights and Ideas
- More Context Makes It Worse: Intuitively, one might think more data helps a model figure out the true signal. Instead, larger context windows actually sharpen the model's commitment to the dominant in-context signal, amplifying spurious routing by up to 1.74× in high-spurious regimes.
- Expressive Models are More Vulnerable: Highly expressive models like TabPFN are generally better at aligning with causal signals. However, in the "high-entanglement corner" (where spurious signals strongly dominate), they are significantly more vulnerable to spurious routing than simple linear models.
- Fixing the Context, Not the Weights: Because ICL models are frozen at inference time, standard domain generalization techniques (which modify training losses or weights) cannot be used. Mitigations must happen entirely within the context window.
Key Results
- Theoretical Validation: The derived closed-form characterization of CSR perfectly matched empirical results, with a Pearson correlation of $r=0.997$ for linear ICL and $r=0.979$ for TabPFN.
- S-Swap Performance: S-swap augmentation reduced spurious routing (CSR) by 74% for linear ICL and an astonishing 98.8% for TabPFN.
- Causal Rerouting: Under S-swap, TabPFN didn't just become agnostic; its causal sensitivity increased by 8.4×, successfully rerouting its prediction mechanism through the true causal signal $C$.
Limitations and Future Work
- Orthogonality Assumption: The current theoretical proofs assume that causal and spurious signals occupy strictly orthogonal subspaces, though empirical tests showed graceful degradation when subspaces overlapped.
- Multi-Environment Requirement: Both proposed mitigations require data from at least two distinct environments (e.g., different hospitals or batches) to work; they collapse to standard ICL in single-environment settings.
- Future Directions: The authors suggest extending the theory to non-linear ICL and exploring the use of CSR as a self-supervised training objective to learn clean representations directly.
Summary updated