Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners

Athanasios Vlontzos, Giorgos Papanastasiou, Bernhard Kainz, Sotirios TsaftarisarXiv 2026 · arXiv

TL;DR

Tabular In-Context Learners (ICLs) like TabPFN can fail silently by relying on spurious signals hidden within composite feature representations (e.g., equipment artifacts bundled alongside real health data). This paper proves that this "spurious routing" is theoretically unavoidable and is actually worsened by larger context sizes or more expressive models. To fix this, the authors introduce "S-swap augmentation," a lightweight, training-free context manipulation that reduces spurious routing by up to 98.8%.

What Problem Does This Paper Address?

In-context learning (ICL) allows models to adapt to new tabular tasks at inference time using just a few labeled examples passed into the context window. However, this flexibility has a major blind spot.

If a single measured feature $X$ bundles a true causal signal $C$ (e.g., a patient's actual health status) with a spurious signal $S$ (e.g., a systematic artifact from a specific hospital's equipment), the model cannot inherently tell them apart. If $S$ correlates with the outcome $Y$ via an unobserved confounder $U$ (like patient demographics) within the training environment, the model will rationally but incorrectly route its predictions through $S$ instead of $C$.

When deployed in a new environment where the equipment artifact changes, the model fails silently. The authors formalize this failure mode as spurious routing in composite representations.

Structural Causal Model of Spurious Routing

Methods

The authors introduce the Causal Sensitivity Ratio (CSR), a perturbation-based diagnostic metric that quantifies whether a model is more sensitive to causal or spurious variations. They mathematically analyze ridge ICL (a linear model baseline) and empirically test TabPFN (a state-of-the-art transformer-based tabular ICL).

To mitigate spurious routing without retraining the frozen models, they propose two lightweight, context-level interventions that only require weak environment labels:

  1. Environment-Stratified Context Construction: Sampling context examples evenly across different environments to break the localized $S–Y$ correlation.
  2. S-swap Augmentation: Swapping the spurious components ($S$) of context examples with those drawn from different environments while keeping the label ($Y$) fixed. This explicitly signals to the model in-context that $S$ does not reliably predict $Y$.

Datasets

  • Synthetic SCM Data: A controlled structural causal model where dimensions of $X$ are explicitly partitioned into causal, spurious, and noise subspaces to validate theoretical predictions.
  • scIB Human Pancreas Dataset: A real-world single-cell genomics dataset containing 16,382 pancreatic islet cells across 5 sequencing technologies (acting as environments) with real batch effects. A controlled confounding path was injected to test the methods under realistic, unknown-$S$ conditions.

Key Insights and Ideas

  • More Context Makes It Worse: Intuitively, one might think more data helps a model figure out the true signal. Instead, larger context windows actually sharpen the model's commitment to the dominant in-context signal, amplifying spurious routing by up to 1.74× in high-spurious regimes.
  • Expressive Models are More Vulnerable: Highly expressive models like TabPFN are generally better at aligning with causal signals. However, in the "high-entanglement corner" (where spurious signals strongly dominate), they are significantly more vulnerable to spurious routing than simple linear models.
  • Fixing the Context, Not the Weights: Because ICL models are frozen at inference time, standard domain generalization techniques (which modify training losses or weights) cannot be used. Mitigations must happen entirely within the context window.

Key Results

  • Theoretical Validation: The derived closed-form characterization of CSR perfectly matched empirical results, with a Pearson correlation of $r=0.997$ for linear ICL and $r=0.979$ for TabPFN.
  • S-Swap Performance: S-swap augmentation reduced spurious routing (CSR) by 74% for linear ICL and an astonishing 98.8% for TabPFN.
  • Causal Rerouting: Under S-swap, TabPFN didn't just become agnostic; its causal sensitivity increased by 8.4×, successfully rerouting its prediction mechanism through the true causal signal $C$.
Table 6: TabPFN at high-spurious corner (ρS=0.7, ρC=0.3, α=1.0).

Limitations and Future Work

  • Orthogonality Assumption: The current theoretical proofs assume that causal and spurious signals occupy strictly orthogonal subspaces, though empirical tests showed graceful degradation when subspaces overlapped.
  • Multi-Environment Requirement: Both proposed mitigations require data from at least two distinct environments (e.g., different hospitals or batches) to work; they collapse to standard ICL in single-environment settings.
  • Future Directions: The authors suggest extending the theory to non-linear ICL and exploring the use of CSR as a self-supervised training objective to learn clean representations directly.

Summary updated

Abstract

Consider a model trained at a single hospital to predict patient recovery, where the measured feature $X$ bundles the patient's true health signal ($C$) with a systematic artefact from that hospital's equipment ($S$). Within that hospital, the artefact correlates with outcomes through unmeasured confounders such as patient demographics; an in-context learner rationally routes predictions through $S$, not $C$, and fails silently when deployed at a new hospital with different equipment. We formalise this as \emph{spurious routing in composite representations}: when a feature $X = [C;\,αS;\,η]$ encodes a causal signal $C$ and a spurious signal $S$ in distinct subspaces, the ICL cannot determine which drives predictions. We prove that under ridge ICL, a linear in-context learner, this routing is unavoidable regardless of context size; TabPFN, a state-of-the-art pretrained tabular ICL model, shows qualitatively consistent behaviour empirically. We derive a closed-form characterisation, $\mathrm{CSR} \propto ρ_S/ρ_C$, confirmed at $r = 0.997$ for linear ICL and $r = 0.979$ for TabPFN. Contrary to intuition, larger context sharpens commitment to the dominant in-context signal, amplifying spurious routing by up to $1.74\times$; in the high-spurious corner, more expressive models show greater vulnerability empirically ($+2.22$ CSR gap at high entanglement). We introduce two lightweight mitigations: environment-stratified context construction and S-swap augmentation, that require only weak environment labels and no knowledge of the causal partition. S-swap reduces spurious routing by $74\%$ for linear ICL and $98.8\%$ for TabPFN, with TabPFN's causal sensitivity increasing $8.4\times$ simultaneously: the model does not become agnostic, it reroutes through the causal signal.
BibTeX
@misc{vlontzos2026entangled,
  author = {Athanasios Vlontzos and Giorgos Papanastasiou and Bernhard Kainz and Sotirios Tsaftaris},
  title = {{Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners}},
  year = {2026},
  eprint = {2607.25532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.25532},
}
Figure