Real tables have holes. Clinical records skip tests, sensors drop out, and survey answers go blank. The few models that cope well with this are large in-context tabular models: transformers that read a labelled sample of the dataset at prediction time and condition on whatever cells happen to be present. These models are accurate, but every prediction pays for attention over the whole support set. Small, fast networks are cheap, but they usually handle missing values by filling them with a column mean or by learning default branches in a tree. Both tricks only know the missingness patterns seen during training.
We propose to close this gap with knowledge distillation, meaning training a small "student" network to copy the output probabilities of a large "teacher." The teacher here is an in-context tabular model that handles missing cells natively. The key move is to query it not only on each training row as recorded, but also on many deliberately sparser versions of that row, with extra cells hidden. Each answer is the teacher's estimate of the label probability given only the cells it can see. The student, which receives cell values plus a mask of which cells are present, is trained to reproduce those answers across masks. In effect the student is taught how predictions should change as information disappears. Nothing in this pipeline ever needs a complete row, so the method can be used on fragmented cohorts where every existing distillation-for-missing-data method breaks down.
Core Claim
The closest prior work distills from a teacher that saw complete data into a student that sees incomplete data. Probabilistic Conformal Distillation maps a partial-modality input to a Gaussian in latent space around complete-modality teacher embeddings. ProtoKD trains a separate single-MRI-sequence student per sequence against a four-sequence teacher. Dis2DR treats complete audio, video and text as privileged training information.
PCD identifies the central difficulty correctly. When information is truly missing, the complete teacher's embedding is not a recoverable target, and forcing a point match makes the student invent detail. Its fix is to widen the target into a Gaussian. Our claim is that the cleaner fix is to change the target itself. If the teacher conditions on exactly the same observed cells as the student, its output is an estimate of the label distribution given those cells. That quantity is well defined for every mask, so matching it asks the student for nothing it cannot in principle know.
We expect two consequences, and both are testable. First, a student distilled across many masks should keep most of the teacher's accuracy at a small fraction of its inference cost, with no support set kept around at deployment. Second, it should degrade more gracefully than tree ensembles or augmentation-only networks when the test-time missingness pattern differs from training, for example when a feature that was always recorded starts going missing. This shift is where learned default split directions and mean filling have nothing to fall back on.
This is not a claim that we beat the teacher. The question is whether the teacher's way of handling missing information survives compression into a model with no attention and no context.
Research Program
The teacher. We use EXAONE Tabular 1.0. Its 20.8M-parameter Cross-Axis Summary Transformer alternates attention across the columns of a row and across the rows of a column. It was pretrained only on synthetic structural causal models and accepts missing cells directly, so the teacher never requires complete rows: its support set is the incomplete training data as recorded. If EXAONE weights are not available, any in-context tabular model with native missing-value handling can stand in. The file names no alternative, so that substitution would need to be reported as a deviation.
The transfer set. For each training row with recorded mask $m$, we draw $K$ additional masks $m' \le m$, which can only hide further cells. We draw them from three sources. The first is random cell drops at varied rates. The second is whole-feature drops that imitate sensor failure. The third is masks borrowed from other rows, which mimic realistic co-missingness. The teacher is run once per (row, mask) pair, offline.
The student. The student is a small multilayer perceptron, or alternatively an additive model in the spirit of Emb-GAM, which showed that a linear additive model fit on frozen foundation-model embeddings can match a large model's predictions at about 1,000x lower latency. The student takes the zero-filled values and the mask as input and minimizes
$$ \mathcal{L} = \mathbb{E}_{x,\,m'}\Big[\mathrm{KL}\big(p_T(y \mid x_{m'}, \mathcal{D}) \,\|\, p_S(y \mid x \odot m', m')\big)\Big] + \lambda\, \mathrm{CE}\big(y, p_S(y \mid x \odot m, m)\big). $$
Here $p_T$ is the teacher's predictive distribution given the cells kept by $m'$ and the support set $\mathcal{D}$, and $p_S$ is the student's prediction. The final term is cross-entropy against the true label $y$, applied only on the recorded mask $m$, and $\lambda$ weights it.
What is new here. The individual pieces exist already. The new ingredient is the sampled family of sparser masks paired with teacher answers for each one. This is what lets a small network learn predictions for missingness patterns that never occur in the raw data.
Analyses. Beyond accuracy, we measure three things. The first is calibration, since the teacher's soft labels should carry its uncertainty about missing cells. The second is whether the student's prediction moves sensibly as features are removed one at a time. The third is how many masks per row are needed: we sweep $K$ over 1, 4, 16 and 64.
The MNAR caveat. Under missing-not-at-random data, the recorded mask is itself informative. Extra masks we impose are random, so they carry no such information. The student sees both kinds through the mask input. We treat this as an empirical question rather than assume it away: we compare training on natural masks only, artificial masks only, and the mixture, under the MNAR settings described below.
Baselines
Naive reference. An MLP trained on mean-imputed inputs plus a mask indicator, using hard labels only. This is the default practice. MissDiff quantifies the cost of mean imputation on MIMIC4ED regression: downstream RMSE of 2.166 for mean-imputed data versus 1.826 for MissDiff-synthesized data. This baseline tells us whether anything beyond trivial handling is needed.
Tuned gradient-boosted trees (XGBoost and CatBoost). These are trained on the raw incomplete table with native missing-value handling. They are the strongest cheap competitor: they are fast, need no teacher, and are the usual tabular choice. The bar they set for the teacher is EXAONE's reported win rate of at least 89% against tuned tree baselines across TabArena and TALENT splits. We have found no reported result on Lacuna for tree ensembles versus EXAONE specifically under MCAR, MAR or MNAR masks on the RefiDiff datasets, so that comparison is ours to run. The trees get the same features and the same hyperparameter search budget, measured in trials and wall-clock time, as the student. The student beats them if it matches or exceeds their AUROC or accuracy in distribution and wins clearly under test-time missingness shift, at comparable CPU latency.
The teacher itself (EXAONE in context). The teacher is the accuracy ceiling. On cost, the paper reports that it matches the 1.64B-parameter TabFM at 78x smaller size and 1/11 of the inference cost. Success means retaining most of the teacher's accuracy gap over trees while cutting per-row latency by at least an order of magnitude. If the student falls back to tree-level accuracy, the distillation has failed.
Impute-then-distill. We fill missing cells with MissForest or with a diffusion imputer (RefiDiff, which reports first- or second-best imputation against DIFFPUTER, ReMasker and HyperImpute), then distill standard logits from the teacher run on the completed rows. This is the strongest "use the existing missing-data machinery" competitor. On Census, MissDiff reports imputation RMSE of 0.112 for MissForest versus 0.087 for itself, showing that the imputation step carries real error. Both pipelines use the same student architecture and the same number of teacher queries.
Two ablations isolate the key ingredient. The first distills only on recorded masks ($K=0$), testing whether the extra masks matter. The second applies the same mask augmentation but trains on hard labels with no teacher, testing whether the teacher matters at all. Each must lose to the full method, especially under missingness shift, for the core claim to hold.
Why not just…?
Why not just use gradient-boosted trees, which already handle missing values? This is the most serious threat. On tables with modest missingness, trees may well match a small distilled network, and EXAONE's 89% win rate is for the teacher, not for any student. Trees route missing values down a default branch that was learned only from patterns present in training. When a usually-recorded feature disappears at test time, that branch was fit on few or no examples. The missingness-shift experiment settles this directly. If trees match the student there as well, the proposal reduces to "trees are enough," and we would report exactly that.
Why not just train the student with random masking and hard labels, without a teacher? This tests whether the teacher is needed at all. Hard labels say what the answer was given the full row. They say nothing about how confident one should be once a key feature is hidden, so the student must infer the right degree of hedging from scratch. The teacher's soft labels provide that hedging directly. The no-teacher ablation, compared on calibration error and on accuracy at high drop rates, is the decisive check. If it ties the full method, the teacher is unnecessary, and the contribution narrows to the masking scheme.
Why not just impute with a strong generative model and then distill normally? Single imputation commits to one guess for each missing cell. Prior evidence shows this compounds error: MissDiff reports SDMetrics fidelity of 51–76% for impute-then-generate pipelines versus about 83% for training directly on masked data on Census. RISE argues that decoupling imputation from inference drifts posteriors because the imputer optimizes reconstruction rather than the target. Diffusion imputers also add multi-step sampling at inference time. We nonetheless run this pipeline as a baseline rather than dismiss it.
Why not just deploy the teacher? At small scale, that is the right answer. The case for distillation rests on deployment settings where keeping a support set and attending over it per query is too slow or too memory-hungry. We report the latency ratio explicitly so readers can judge where the trade-off flips.
Proposed Experiments
Main benchmark. The nine-dataset RefiDiff suite (California, Magic, Bean, Gesture, Letter, Default, News, Adult, Shoppers), restricted to the tasks with a prediction target. We follow its protocol of MCAR, MAR and MNAR masks with 10 random draws. Metrics are accuracy or AUROC for classification, RMSE for regression, expected calibration error, and per-row CPU and GPU latency.
Extreme missingness and a real cohort. We add the BCCO benchmark, used in the EXAONE report, for extreme native missingness. We add MIMIC-IV mortality prediction, built with the MIMIC-IV processing pipeline, which reports AUROC 0.87 for a hybrid LSTM versus 0.67 for logistic regression on its kidney-disease ICU cohort. Those numbers are reference points, not directly comparable baselines, because our inputs are static tables. MIMIC-IV gives a cohort where complete rows are rare, so the complete-case teachers assumed by PCD, ProtoKD and Dis2DR would discard most of the data. We report what fraction of rows is complete to make that concrete.
The decisive test: missingness shift. We train on one mechanism and rate, for example 20% MAR. We then test on three shifted conditions: 50% MCAR, MNAR, and entire held-out features removed. We compare the curve of accuracy against test-time drop rate for the student, trees, the no-teacher ablation and the teacher. The hypothesis is supported if the student's curve tracks the teacher's and stays above the trees and the ablation as missingness grows, while the student is at least ten times faster than the teacher.
Mask budget. A sweep over $K$ shows how many teacher queries the benefit requires, which bounds the offline cost.
Fairness check. Following Chouldechova et al., who show that missing values silently coded as zeros can bias models against protected groups, we report subgroup error rates on MIMIC-IV. A student that copies the teacher may also copy its biases.
References
- Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness
- Prototype Knowledge Distillation for Medical Segmentation with Missing Modality
- Disentangled-Multimodal Privileged Knowledge Distillation for Depression Recognition with Incomplete Multimodal Data
- EXAONE Tabular 1.0 : Technical Report
- Emb-GAM: an Interpretable and Efficient Predictor using Pre-trained Language Models
- MissDiff: Training Diffusion Models on Tabular Data with Missing Values
- RefiDiff: Progressive Refinement Diffusion for Efficient Missing Data Imputation
- Robust Simulation-Based Inference under Missing Data via Neural Processes
- An Extensive Data Processing Pipeline for MIMIC-IV
- The Impact of Differential Feature Under-reporting on Algorithmic Fairness