When one class makes up 5% of a dataset, a standard classifier can score well by almost always predicting the common class. The usual fix is to throw away most of the common-class examples until the classes are balanced. This works surprisingly well, but it deliberately discards data. On a cohort of 100 patients, that can leave a model with about ten examples per class. This proposal asks a sharp question in the setting where the waste hurts most: small tabular datasets classified by in-context tabular models. These are pretrained transformers, such as TabPFN, that read the whole training set as a prompt and output predictions without any weight updates. Does downsampling help only because it shifts the model's built-in belief about how common each class is? If so, moving the decision threshold would recover the entire benefit for free. Or does it also change which decision boundary the model draws?
Our claim is that the benefit splits into those two parts, and that the second part can be kept without paying the data cost. The method is to query the model many times, each time with a different balanced random subset of the majority class (and all minority points). Each answer is corrected for that subset's artificial class balance, and the answers are averaged. Every training point gets used, each query is balanced, and the extra cost is a handful of forward passes. If this "balanced context bagging" beats a tuned threshold on the full data, downsampling does something thresholding cannot. If it does not, practitioners can stop downsampling small tabular datasets altogether. Either outcome settles a question the literature currently leaves open.
Core Claim
Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification (also here; Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification) is the closest work and the one this proposal is built around. It compares four fixes for imbalance in TabPFN on 11 binary OpenML-CC18 datasets: shifting the threshold to the minority proportion, randomly downsampling the context, duplicating minority points, and generating synthetic points. Threshold shifting gets the top average rank. Random downsampling also strongly raises minority accuracy, while synthetic oversampling hurts. That paper uses one random downsample per prediction, studies only binary tasks, and never asks why thresholding and downsampling both work.
We separate two effects. The first is a prior effect: downsampling changes the class frequencies the model sees, which a calibrated model turns into a shifted posterior. Thresholding replicates this exactly. The second is a boundary effect: with fewer majority tokens dominating attention, the model may place the boundary differently near the minority class. Thresholding cannot replicate this. The variance cost of downsampling is real in small samples. Averaging over many balanced draws, each corrected back to a common prior, should remove most of that cost while keeping whatever boundary effect exists.
Concretely, let $\pi_y$ be the class frequencies in the full training set and $\pi^{(k)}_y$ the frequencies in balanced draw $k$. Let $p_k(y\mid x)$ be the model's posterior when prompted with draw $k$. Each draw is reweighted by the standard prior-shift correction, and the corrected posteriors are averaged over $K$ draws:
$$\bar p(y\mid x) \;\propto\; \frac{1}{K}\sum_{k=1}^{K} \frac{p_k(y\mid x)\,\pi_y/\pi^{(k)}_y}{\sum_{y'} p_k(y'\mid x)\,\pi_{y'}/\pi^{(k)}_{y'}}.$$
A single decision rule, whether a balanced-error threshold in the binary case or a per-class threshold vector in the multi-class case, is then applied to $\bar p$. The design makes the decomposition testable. Without the boundary effect, $\bar p$ equals the full-context posterior up to noise, and the two thresholded classifiers tie.
Why this framing, given prior work
Theory on this question points in two directions. In random-feature regression, How does overparametrization affect performance on minority groups? proves that subsampling the majority strictly lowers minority risk. In classification, An Investigation of Why Overparameterization Exacerbates Spurious Correlations shows that subsampling to 3.4% of CelebA cuts worst-group error to 15.1%, versus 25.6% for full-data reweighting. Together these papers motivate the claim that downsampling does more than reweight. Both, however, work with large datasets and trained weights, not frozen in-context models on a few hundred rows.
The Group Robustness is in the Details shows that upsampling during finetuning collapses, falling to about 50% worst-group accuracy on Waterbirds. Its partial-subsetting fix, Mixture Balancing, reaches about 81.1%, but its subset ratio is a hand-tuned knob. That paper constrains our design: we exclude duplication outright and treat the subset size $m$ as something to study rather than tune by hand.
MESA learns a sampler that picks majority subsets for successive tree-ensemble members. It is the nearest relative of bagging over balanced subsets. It needs a separate clean validation split to compute its reinforcement-learning reward, which small cohorts cannot afford, and it trains new estimators. Our method trains nothing and reserves no data.
Finally, DRoP argues that subsampling within a class should be random so the class-conditional distribution is preserved. Severing Spurious Correlations with Data Pruning argues for removing the hardest examples deterministically. We test this disagreement inside each draw. In-context models give each training point an almost free leave-one-out score: we predict it from a prompt that excludes it. Difficulty-aware selection therefore needs no validation split.
Research Program
The main artifact is a selection-and-aggregation wrapper around frozen in-context tabular models. TabPFN is the primary model, and any public successor with the same interface would be a secondary one. The wrapper has two parameters: the number of draws $K$ and the majority subset size $m$, expressed as a ratio to the minority count. In the multi-class case, each draw caps every class at a quota $m$ and keeps all points from classes below the quota. The correction above handles arbitrary class frequencies, so it extends to the multi-class setting that McDowell et al. leave open.
The analysis has three parts. First, we measure how far the bagged, corrected posterior $\bar p$ moves from the full-context posterior, using total-variation distance on test points. We break this distance down by distance to the minority class. A boundary effect should concentrate near minority points, while pure noise should be uniform. Second, we trace performance as $m$ goes from fully balanced to the full majority count, for each $K$. This replaces LaBonte et al.'s hand-tuned ratio with a measured curve. Third, we compare random within-class selection against leave-one-out-difficulty selection, in both directions: keeping the hardest points and dropping them. This gives DRoP's and Severing's principles a direct comparison in a regime where dropping hard points could delete the only atypical minority-adjacent examples.
Baselines
The naive reference point is TabPFN on the full imbalanced context with its default decision rule and its default internal ensembling over feature and class permutations. McDowell et al. use this as their base model. At a minority fraction of 0.05, they report that thresholding and downsampling raise minority accuracy by 40–60% over it.
The strongest existing competitor is threshold shifting on the full context, with the threshold set to the empirical minority proportion. McDowell et al. found it achieves the top average rank across their 11 OpenML-CC18 datasets. The file does not contain a per-dataset worst-class or balanced-accuracy value for it ("no reported result found on Lacuna for TabPFN threshold-shifting worst-class accuracy per dataset"), so we will reproduce it on identical splits before any comparison. We also give it its strongest form: a threshold chosen by in-sample leave-one-out balanced accuracy, which uses no extra data. Beating it means a higher mean balanced accuracy and worst-class accuracy across datasets and seeds, at a pre-registered paired-test significance level. We will also report every case where it ties.
The second baseline is single random downsampling, McDowell et al.'s second-best method. It isolates what averaging adds.
The key ablation equalizes compute. It runs $K$ full-context TabPFN passes under different permutations, which is ensembling without balancing, and thresholds the averaged result. It gets exactly the same number of forward passes as the bagged method. If balanced bagging only matches this ablation, the gain came from ensembling, not from balance.
A second ablation removes the per-draw prior correction. This shows whether the correction is necessary or whether balanced draws should simply be averaged and then thresholded.
For a non-in-context reference, we run gradient-boosted trees with class weighting and with balanced bagging. We report MESA wherever enough data exists to hold out its validation split. MESA's reported result is an AUCPRC gain of up to 24.7% over the next best method on Mammography with small ensembles. We reproduce its setting on Mammography, where we report AUCPRC to match.
All methods see the same training rows, the same test rows, and no extra validation data. Every tunable choice, including $m$ and the threshold, is made by in-sample leave-one-out or nested cross-validation inside the training rows.
Why not just…?
Why not just shift the threshold, since it already wins? This is the real threat, and the main experiment is designed around it. If the boundary effect is negligible, thresholded full-context TabPFN will tie balanced bagging. We would then report that downsampling small tabular data should be abandoned in favor of thresholding, a useful negative result. Two pieces of evidence argue for a boundary effect. Sagawa et al.'s subsampling result in trained networks is one; there, reweighting, which is roughly the training-time analogue of a prior correction, fails while subsampling succeeds. The other is McDowell et al.'s finding that downsampling alone, without any threshold change, still moves minority accuracy substantially. Whether this carries over to frozen in-context models is exactly what the posterior-distance analysis tests. Thresholding's advantage is also least established in the multi-class case, where we evaluate the threshold-vector version directly.
Why not just bag balanced subsets with trees, which is old? Balanced bagging for tree ensembles is a known classical idea. We do not claim the averaging trick as new. We include tree-based balanced bagging and MESA as baselines, and if they beat in-context methods on small tables, we will say so. What is new is the question of whether balancing changes a frozen in-context model's boundary beyond a prior shift, and the prior-correction step that makes the answer measurable. Trees retrained on each subset tangle the two effects, because each member's structure changes. A frozen in-context model isolates them.
Why not just oversample the minority class instead? McDowell et al. found that duplicating minority rows in the context degrades TabPFN and that TabPFGen synthesis performs worse than no augmentation. LaBonte et al. found that upsampling collapses to about 50% worst-group accuracy on Waterbirds. We include context duplication as a baseline for completeness, but the existing evidence already rules it out as a design.
Why not just rely on TabPFN's built-in ensembling? That is exactly the compute-matched ablation above. If permutation ensembling plus a threshold equals balanced bagging, balance contributes nothing.
Proposed Experiments
The main benchmark is the 11 binary OpenML-CC18 datasets used by McDowell et al., including adult, spambase, kr-vs-kp, and bank-marketing. Minority fractions are $\pi_1 \in \{0.05, 0.1, 0.2\}$, induced by subsampling the minority class. Total training sizes are $N \in \{50, 100, 200, 500\}$, with 20 seeds per cell. The small-$N$ axis is the part McDowell et al. did not vary and the part this proposal is about. At $N=100$ and $\pi_1=0.05$, a single balanced draw has ten rows, so the variance cost of downsampling should be most visible there.
For multi-class, we induce long-tailed class frequencies on the CC18 multi-class tasks that fit TabPFN's class limit, and we report worst-class and balanced accuracy. Mammography provides a real imbalanced comparison against MESA. Metrics are balanced accuracy, worst-class accuracy, AUCPRC, and expected calibration error of $\bar p$ after correction.
Three checks are decisive. The first is the paired comparison of balanced bagging against the best-tuned threshold on the full context, broken out by $N$. Our prediction is that the gap is largest at intermediate $N$. At very small $N$, every draw is too thin. At large $N$, the boundary effect fades as the majority no longer crowds out minority evidence. The second is whether posterior differences concentrate near minority points. This separates a boundary effect from noise and is predicted to hold if the first check succeeds. The third is the random-versus-difficulty selection comparison at $N \le 200$. If hardest-first pruning wins on large $N$ but loses on small $N$, that reconciles DRoP and Severing Spurious Correlations by sample size rather than by principle. If thresholding ties bagging throughout, the paper's conclusion becomes that, for calibrated in-context tabular models, downsampling is a threshold in disguise.
References
- Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification
- How does overparametrization affect performance on minority groups?
- An Investigation of Why Overparameterization Exacerbates Spurious Correlations
- The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations
- MESA: Boost Ensemble Imbalanced Learning with MEta-SAmpler
- DRoP: Distributionally Robust Data Pruning
- Severing Spurious Correlations with Data Pruning