A robust model usually loses some accuracy on clean data. The standard explanation is that one fixed predictor has to hedge against corruption everywhere, even on inputs that are never corrupted. In-context Bayesian learners do not work that way. A prior-data fitted network (PFN) is a transformer trained offline on millions of synthetic datasets so that, given a new dataset as its prompt, it outputs an approximate Bayesian posterior prediction in one forward pass. TabPFN is the best-known example (TabPFN). Such a model could, in principle, read the context, infer whether that context shows signs of corruption, and hedge only when it does. We propose to build and test this. We pretrain a tabular PFN on a prior that includes an explicit contamination process, with a user-settable contamination prior rate supplied at inference time. We then measure whether its clean-data accuracy loss shrinks as clean context grows, while its robustness to corrupted context stays intact.
The target is the setting that current work leaves open: corrupted or poisoned context. This covers mislabeled rows, outlier feature values, and adversarially crafted training points, all of which are common in messy real tables. The claim is specific. For corruption that leaves statistical evidence in the context, the robustness-accuracy trade-off is not fixed. It is a quantity set by the posterior probability of contamination, and it decays with context size.
Core Claim
Suppose the pretraining prior is a mixture: clean tasks with weight $1-\rho$, and contaminated tasks with weight $\rho$, where a latent process corrupts a fraction of context rows. The Bayes-optimal predictor $q_\rho$ under this prior satisfies, on clean tasks,
$$ R_{\text{clean}}(q_\rho) - R_{\text{clean}}(q_0) \;\le\; C \,\mathbb{E}_{D \sim \text{clean}}\big[\pi_\rho(\text{contaminated} \mid D)\big]. $$
Here $R_{\text{clean}}$ is expected loss on clean tasks, $q_0$ is the Bayes predictor under the clean-only prior, $\pi_\rho(\cdot \mid D)$ is the posterior over the contamination regime given context $D$, and $C$ bounds the loss difference between the two regimes' predictions. The bound follows because $q_\rho$ is a posterior-weighted mixture of the two regime-specific predictors.
When clean and contaminated regimes differ in likelihood, the posterior on contamination given clean data typically shrinks quickly with the number of context rows $n$. Under standard posterior concentration it shrinks roughly exponentially. So the clean tax should fall with $n$ and rise with $\rho$. A fixed robust estimator, by contrast, pays a constant tax.
We will test whether a trained PFN actually realizes this adaptive behavior. It may not, because it only approximates the Bayes predictor, and the bias plateau reported in Statistical Foundations of Prior-Data Fitted Networks is a warning.
This framing also accounts for a negative result that seems to contradict it. Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners adversarially pretrains a single-layer linear transformer against perturbations of the query point, with clean demonstrations, and finds that the trade-off persists (99% vs. 100% clean accuracy on synthetic data). That paper's regime is exactly the one where the bound gives no relief. A worst-case perturbation of one query leaves no trace in the context, so the model cannot infer when to hedge. We take that result as a constraint and restrict the claim to corruption that is identifiable from context. This is the setting that paper explicitly excludes and that the open-environment failures of TabPFN v2.5 point to (Realistic Evaluation of TabPFN v2.5 in Open Environments).
How This Differs From Existing Robust Tabular Foundation Models
Three lines of work already apply adversarial ideas to tabular PFNs, but all of them target prior coverage rather than corruption in the context itself. APT trains small generator networks to produce synthetic tasks that maximize the transformer's loss. RTFM searches generator parameters where TabPFN V2 trails tree ensembles most, then continues pretraining there. RAD-TFM adds optimal-transport alignment to target domains.
None of the three corrupts the context, evaluates poisoning, or measures the clean-data cost of robustification. The research file flags this last omission as an open gap. Our method reuses their continued-pretraining recipe of roughly 100k synthetic datasets from a TabPFN V2 checkpoint. It changes what the synthetic tasks contain and what the model is asked to output.
Research Program
The contamination prior. We extend TabPFN's structural-causal-model prior with a latent contamination stage. Per task, we draw a regime indicator with probability $\rho$ and a contamination fraction. When contaminated, we apply one of several corruption families to a subset of context rows:
- random and class-conditional label flips;
- heavy-tailed feature outliers;
- rows drawn from a different causal model, which mimics concept shift;
- "plausible" poisons generated by a second causal model sharing the feature marginals.
The query rows always come from the clean task. The model is therefore trained to predict the clean conditional distribution while seeing contaminated evidence.
The prior-rate token. Following the on-the-fly prior adaptation idea in Distribution Transformers, which feeds the prior itself as input tokens, we sample $\rho$ per training batch and feed it as a conditioning token. A single model then approximates $q_\rho$ for a range of $\rho$. At deployment, a practitioner can set how much robustness to buy and read off the resulting clean tax.
The auxiliary contamination head. The model also predicts two quantities: a per-context-row contamination probability and the task-level regime posterior. This makes the model's internal belief about contamination measurable. The central diagnostic is whether the task-level posterior on clean data actually decays with $n$ and whether the clean tax tracks it.
The adaptive attack. Because context rows are continuous inputs to a differentiable network, we can run gradient-based poisoning. The attacker perturbs or inserts a budgeted number of context rows to maximize query loss, with full knowledge of the model and of the contamination head. The attacker can therefore try to craft poisons the model believes are clean. This is the test that separates a genuine Bayesian adaptation from a detector that is easily fooled.
Baselines
The naive reference point is unmodified TabPFN V2 given the corrupted context as-is. It shows how much damage each corruption family does and fixes clean accuracy at the pretrained level. For clean performance on TabPertNet, the file reports TabPFN V2 at 0.7306 mean normalized AUC (RAD-TFM).
The strongest existing tabular foundation model on clean benchmarks is RAD-TFM, at 0.8486 mean normalized AUC on TabPertNet. Its predecessor RTFM reports up to a 6% gain in mean normalized AUC over TabPFN V2 on TabPertNet and TabArena (RTFM). We found no reported result on Lacuna for any tabular foundation model under context poisoning or label contamination, so robustness numbers for all methods will be measured fresh.
We run RTFM's optimality-gap continued pretraining ourselves, with the same checkpoint, the same 100k-dataset budget, and the same optimizer settings as our method. Our model also receives 100k datasets. Beating RTFM means two things together. First, clean-benchmark AUC must be within noise of RTFM on TabPertNet and TabArena. Second, AUC on contaminated versions of the same datasets must be significantly higher.
The strongest classical competitor is filtering before prediction. We apply the localized $k$-NN context selection of Statistical Foundations of Prior-Data Fitted Networks, or an isolation-forest outlier filter, to the context and then pass the survivors to TabPFN V2. The filter threshold is tuned on the same validation corruptions our model sees in pretraining.
We also include CatBoost and XGBoost, tuned under the same corruptions, since trees were more robust than TabPFN v2.5 under some shifts (Realistic Evaluation).
The key ablation is fixed-rate corruption augmentation. It uses the same contamination prior at one fixed $\rho$, with no prior-rate token and no auxiliary head, trained on the same 100k datasets. It isolates whether conditioning and explicit contamination inference matter beyond simply seeing corrupted training tasks. A second ablation removes only the auxiliary head.
Why not just…?
Why not just filter suspicious context rows before feeding TabPFN? Filtering is cheap and is the first thing a practitioner would try. The file's evidence cuts against it in two ways. The $k$-NN localization in Statistical Foundations of Prior-Data Fitted Networks reduces bias at the cost of higher variance, and its author calls it a stopgap. Input-level cleansing in other domains rejects valid but unusual points and directly costs clean accuracy (Training set cleansing of backdoor poisoning; Baseline Defenses for Adversarial Attacks Against Aligned Language Models). A filter also cannot catch label flips on inlying rows, because a row's features look normal even when its label is wrong. Our experiments report filter performance separately for each corruption family. If filtering matches our model on feature outliers, we will say so and narrow the claim to label and concept contamination.
Why not just continue pretraining TabPFN on corrupted tasks at a fixed rate? This is the real threat, because it may capture most of the gain. The Bayesian argument predicts one specific difference. At any fixed rate, the model's clean tax is set once at training time. With the prior-rate token and contamination head, the clean tax should shrink faster with $n$ and be controllable after training. The ablation settles this directly. If the fixed-rate model's clean-tax-versus-$n$ curve and its robustness are indistinguishable from ours, then the prior-rate token is unnecessary. The contribution then reduces to the contamination prior plus the measurement of the trade-off, and we will report it that way.
Why not just use gradient-boosted trees with robust losses? Trees are strong and do not need pretraining. However, a robust loss imposes a fixed hedge with a constant efficiency cost on clean data, which is exactly the constant tax we aim to remove. Trees also required over 3,500 seconds of tuning against 0.90 seconds for APT (APT). The comparison is empirical. If tuned robust CatBoost dominates on contaminated data at equal clean accuracy, the proposal fails on its own terms.
Why not just adversarially pretrain against query perturbations, as in the linear-transformer work? That work shows the trade-off persists for query attacks and requires more clean demonstrations to recover clean accuracy (Adversarially Pretrained Transformers). It addresses a threat that leaves no trace in the context, so it cannot adapt its hedge to the data. We therefore treat query attacks as out of scope rather than as a competing solution.
Proposed Experiments
Evaluation uses TabArena and TabPertNet with RTFM's size caps, plus OpenML-CC18 for comparability with TabPFN and APT. Real concept shift comes from TableShift and WhyShift. Each clean dataset is evaluated both unmodified and with each corruption family injected into the context at 5%, 10%, and 20% of rows. The adaptive gradient-based poisoning attack is run at matched row budgets. Metrics are one-vs-one AUC and log loss on clean query rows, and area under the contamination-detection curve for the auxiliary head.
The first decisive check is the clean-tax curve. On clean datasets, we plot AUC loss relative to TabPFN V2 against context size $n$ for several $\rho$ settings. The core claim predicts a decreasing curve that tracks the head's estimated contamination posterior. A flat curve would refute adaptive behavior.
The second check is the Pareto comparison. For each method, we plot clean AUC against contaminated AUC, averaged over datasets. The proposal succeeds only if our model sits on or above the frontier traced by the fixed-rate ablation across its $\rho$ settings, and above RTFM, filtering, and tuned trees.
The third check is the adaptive attack. We report contaminated AUC under poisons optimized against the full model. The key question is whether poisons that fool the contamination head are, as the Bayesian view predicts, also less harmful to query predictions. If they are not, the model has learned a brittle detector rather than a posterior, and that finding would itself be reported as the main negative result.
References
- TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
- Statistical Foundations of Prior-Data Fitted Networks
- Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners
- Realistic Evaluation of TabPFN v2.5 in Open Environments
- Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer
- Robust Tabular Foundation Models
- RAD-TFM: Robust and Domain-Adapted Tabular Foundation Models
- Distribution Transformers: Fast Approximate Bayesian Inference With On-The-Fly Prior Adaptation
- Training Set Cleansing of Backdoor Poisoning by Self-Supervised Representation Learning
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models