Repairing Abstentions in Tabular Foundation Models Without Breaking the Risk Guarantee

Tabular foundation models such as TabPFN take a whole small dataset as their input and predict labels for new rows in one forward pass, with no training. Their probability estimates are good enough that a natural way to make them safe is to let them abstain: answer only when confident, and hand the rest to a human. This abstain-when-unsure step is called selective prediction. It works, but it is passive. The rows the model is unsure about are often exactly the rows where the raw columns hide a relationship its synthetic pre-training never taught it, and no threshold can fix that. This proposal asks whether a language model that writes and debugs feature-engineering code, targeted specifically at the region where the tabular model abstains, can turn a large share of those abstentions into answers. The key requirement is that the error rate on everything the system answers must still be provably bounded.

The central difficulty, and the reason this is more than plugging two existing systems together, is that repair corrupts the confidence signal. If an agent keeps rewriting the features of uncertain rows until the tabular model becomes confident, the new confidence reflects how hard the agent searched, not only how likely the prediction is to be right. The core claim is that a two-stage cascade can escape this trap. In the first stage the frozen model answers confident rows. In the second, abstained rows are re-predicted on a code-repaired representation. Coverage at a fixed error budget can rise only if the repair code is written on one data split and the acceptance thresholds for both stages are certified jointly on a split the language model never saw. The claim also predicts the failure of the obvious alternative: reusing the original threshold after repair should visibly exceed the risk target.

What exists and what is missing

Three strands supply the parts. TabPFN and its successor Accurate predictions on small data with a tabular foundation model show that a transformer pre-trained on synthetic datasets approximates Bayesian posterior prediction and matches tuned tree ensembles on small tables within seconds. Our first stage reuses this model unchanged.

When Data Is Scarce: The Strength of the Prior in Tabular Foundation Models frames the model's behaviour as dominated by its synthetic prior when data are few. This motivates the proposal: uncertainty is likely to concentrate where the true mechanism departs from that prior. A change of representation can move such rows back into territory the prior handles well.

CAAFE is the closest method. It has GPT-4 write pandas feature code from dataset descriptions, repairs crashes by feeding tracebacks back to the model, and keeps a feature if cross-validated TabPFN performance improves. It raised mean ROC AUC from 0.798 to 0.822 over 14 datasets, with gains saturating around ten iterations. We reuse its generate-execute-debug loop. We change what the loop optimizes: the log-loss on the abstained region, subject to not harming the accepted region, rather than global AUC. We also add what CAAFE lacks entirely, which is an abstention decision and a certified bound on the error of the answers the system gives.

Selective prediction supplies the certification. Selective Classification for Deep Neural Networks defines the pair of predictor and accept/reject rule. It shows that maximum-probability thresholding ranks errors well, and it gives a binomial-bound procedure (SGR) that picks a threshold whose risk on accepted cases stays below a target with high probability. Act or Defer shows that marginal guarantees can hide disastrous per-class behaviour. It reports that standard conformal selection let about 90% of positive-class actions be wrong on a rare cancer task, and it fixes this with stratified, class-specific control. That result constrains us: we must report and optionally control per-class risk, not only marginal risk. Its deferrals are terminal, whereas ours are a trigger for repair. Selective Code Generation under Correctness and Security Risk applies conformal risk control to LLM code, reaching 0% observed silent failures at 82.6% coverage versus 42.1% leakage for plain pass@1. It explicitly leaves multi-turn refinement uncovered, and that is the setting we are in.

Finally, Statistical Test for Auto Feature Engineering by Selective Inference is a constraining result. Searching over feature transforms and then evaluating on the same data inflates error rates. This is why our design separates the data used to write code from the data used to certify thresholds.

Core Claim

Let $s_0(x)$ be TabPFN's maximum class probability on the raw features. Let $s_1(x)$ be its maximum probability after applying a repair transform $\phi$ to both the context rows and the query. The transform $\phi$ is a function of features only, written by the agent and applied identically to all rows. The cascade answers with the stage-0 prediction if $s_0(x)\ge\tau_0$. Otherwise it answers with the stage-1 prediction if $s_1(x)\ge\tau_1$, and abstains otherwise.

Because $\phi$ is fixed before calibration, the whole cascade is a fixed function of $x$ once the thresholds are chosen. Calibration and test rows therefore remain exchangeable. Fixed-sequence or Bonferroni testing over a grid of threshold pairs, with SGR-style binomial bounds for each pair, then yields a selected pair satisfying

$$\Pr\big[R(\tau_0,\tau_1)\le\alpha\big]\ge 1-\delta,$$

where $R$ is the error rate among answered test rows, $\alpha$ is the user's error budget, and $\delta$ is the allowed failure probability of the calibration. Among valid pairs we choose the one with the highest coverage.

The empirical claim is that at fixed $(\alpha,\delta)$, this cascade answers substantially more rows than certified thresholding on raw TabPFN. It should also answer more than certified thresholding on globally CAAFE-engineered TabPFN.

Research Program

Each dataset is split four ways: a context split fed to TabPFN, a development split, a calibration split, and a test split.

The first stage runs TabPFN on the development split and marks rows below a provisional threshold as the abstained region. The agent receives the dataset description and schema, as in CAAFE. It also receives a contrastive summary of the abstained region: per-feature distribution shifts between abstained and accepted rows, the class mix, and the most confused class pairs. It then writes candidate transforms in a sandbox with traceback-driven debugging.

A transform is kept if it lowers TabPFN log-loss on the abstained development rows without raising it on the accepted ones. We cap the number of added features so the table stays under TabPFN's width limit. The loop runs for at most ten rounds, matching CAAFE's observed saturation.

The language model never sees the calibration or test splits. Threshold certification happens afterwards. At deployment no LLM is called. Abstained queries pay only for applying a stored transform and one extra TabPFN pass, so latency scales with the abstention rate rather than with agent iterations. This is how the design reconciles the model's second-scale inference with the agent's minute-scale loop.

Three analyses accompany the method. First, we measure confidence inflation directly by comparing the empirical error of stage-1 rows at the reused threshold $\tau_0$ against the target. Second, we check per-class selective risk, following the concern raised by Act or Defer, and test a class-stratified variant of the threshold certification. Third, we audit the repair code for leakage and for "hallucinated logic": features with persuasive rationales and no signal, which CAAFE documents. We do this by checking whether kept features' gains replicate on the calibration split.

Baselines

The do-nothing reference is TabPFN on raw features with maximum-probability thresholding certified by SGR on the same calibration split, following Selective Classification for Deep Neural Networks. It sets the coverage any repair must exceed at each $\alpha$. We found no reported result on Lacuna for coverage at fixed selective risk for TabPFN on OpenML-CC18 or the CAAFE datasets, so we will establish this number ourselves.

The strongest existing method is global CAAFE followed by the same SGR certification. It is the strongest competitor because it uses the same language model, the same debugging loop, and the same downstream model. The only differences are its objective and the absence of a second stage. The only reported figure is its accuracy result, mean ROC AUC from 0.798 to 0.822 on its 14 datasets in CAAFE. We will first reproduce this, then run CAAFE on the context-plus-development data with an identical LLM call budget, iteration cap and feature cap. Beating it means higher certified coverage at $\alpha\in\{0.05,0.10\}$, with a paired test across datasets and repeated splits.

A strong non-LLM local competitor handles abstained rows by rebuilding TabPFN's context from their nearest training neighbours plus a larger permutation ensemble. It is a two-stage cascade whose repair is retrieval rather than code, in the spirit of the LocalPFN comparator reported in TabPFN Unleashed. It is certified with the same two-threshold procedure.

A tuned gradient-boosted tree ensemble with the same threshold certification covers the reviewer who doubts the foundation model is needed.

The key ablations isolate the proposal's ingredients. The region-targeted objective is swapped for CAAFE's global objective inside our cascade. Code debugging is disabled, giving single-shot generation. The LLM is replaced by random compositions of arithmetic operators, like the operator space in the selective-inference paper, which tests whether semantics matter at all. The naive variant reuses $\tau_0$ at stage 1 without joint certification.

Why not just…?

Why not just threshold TabPFN's own confidence? TabPFN's probabilities are good, and for many datasets this may already reach high coverage. It is the reference point, not a straw man. What thresholding cannot do is change which rows are hard. If a mechanism lies outside the prior, as the data-scarcity paper suggests, those rows stay uncertain and the only lever is lower coverage. The experiment settles this per dataset. Where the repair yields no coverage gain, we will report it, and we expect the gains to concentrate on datasets with meaningful column semantics.

Why not just run CAAFE once globally and then threshold? This is the real threat. Global features may fix most of what region-targeted features fix, since any feature that sharpens a decision boundary helps both. If global CAAFE plus certification matches our cascade, the targeting claim fails. What survives in that case is the certified two-stage procedure and the measured confidence-inflation effect. The ablation that swaps objectives inside the cascade separates "targeting helps" from "a second certified stage helps." The global-versus-cascade comparison separates both from plain feature engineering.

Why not just fix hard rows without code, by giving TabPFN local context or more ensembling? Retrieval-based local context is cheaper, involves no LLM, and directly addresses local ambiguity. It could plausibly match code repair on datasets whose uncertainty comes from heterogeneity rather than a missing interaction. It runs under the identical cascade and certification, so the comparison is exact. We expect code to win only where a semantically meaningful derived quantity exists, such as ratios, durations or domain scores. The random-operator ablation checks whether the LLM's semantics, rather than mere feature expansion, carry the gain.

Why not just use gradient-boosted trees with conformal abstention? Trees are fast and can be certified the same way. On small tables TabPFN matches tuned tree ensembles in Accurate predictions on small data with a tabular foundation model, but whether its coverage at fixed risk also matches is untested. We include trees as a baseline, and we also run our cascade with trees as the downstream model, to see whether the repair benefit belongs to TabPFN specifically.

Proposed Experiments

The main benchmark is the 14-dataset CAAFE suite. It has the natural-language descriptions the agent needs, and it includes four Kaggle datasets released after the LLM's training cutoff to limit memorization. We add the small numerical OpenML-CC18 subset used by TabPFN. We also add a stratified sample of TALENT datasets that fit TabPFN's limits, to check that results are not confined to hand-picked tables. For each dataset we use at least 20 random four-way splits.

The metrics are certified coverage at $\alpha\in\{0.05,0.10\}$ with $\delta=0.1$, the area under the risk-coverage curve, and the frequency with which realized test risk exceeds $\alpha$ across splits, which should stay at or below $\delta$. We also report per-class selective risk, LLM tokens and wall-clock time per dataset, and test-time latency per query.

Two checks decide the proposal. The first is the inflation test. The naive cascade that reuses $\tau_0$ should exceed $\alpha$ on a clearly larger fraction of splits than $\delta$, while the jointly certified cascade should not. If the naive version is also safe, the paper's methodological warning is wrong and we will say so. The second is the coverage test against global CAAFE under matched budgets. A consistent paired gain in certified coverage, concentrated on datasets with meaningful columns and absent under the random-operator ablation, supports the claim that uncertainty-targeted code repair adds something that neither global feature engineering nor passive abstention provides.