Pre-Training on the Null Bitmap: Treating Missingness as Signal, Not Tokens

Large tables in specialized domains, such as hospital records, survey data and app event logs, are mostly empty. A patient record has room for thousands of lab tests but holds a few dozen. Database engineers solved the storage side of this long ago. One method stores only the fields that are present. Another, called "bitmap null stripping", splits each column into two parts: a bit vector recording which cells exist, and a dense array holding the actual values. Tabular foundation models discard this solution. These are transformers pre-trained on many tables and then adapted to new ones. They usually turn each row back into a wide sequence, often writing "Feature X: None" for every empty field, or they silently drop empty fields and pre-train as if values were missing at random. The first choice wastes compute in proportion to how wide the schema is. The second discards a strong clinical signal: whether a test was ordered at all.

This note proposes to pre-train the model the same way the storage engine splits a row. Each record enters the transformer as only its present (attribute, value) pairs, so compute grows with the number of filled cells rather than with schema width. The pre-training objective has two parts. One predicts the record's null bitmap, meaning which attributes were recorded. The other predicts the values of recorded attributes given that presence pattern. The bet is that learning to predict presence is what lets a model transfer well in domains where missingness is informative. The test is a controlled experiment in which we switch the missingness mechanism between random and informative.

Core Claim

In wide, sparse domain tables, the pattern of which fields are recorded carries much of the predictive signal. The strongest evidence is from How Should Transformers Encode Numeric Values in Electronic Health Records?, which encodes lab values in a ModernBERT health-record model. It finds that adding numeric results improves disease-onset AUROC by only 0.025 to 0.075, and that the bare presence of a test is almost as predictive as its result. Standard tabular pre-training objectives do not model this. They mask cells and reconstruct values under an implicit assumption that cells are missing at random.

We claim that splitting the objective into a presence term and a value term does two things. It recovers this signal as a transferable representation. It does so at a cost that scales with the number of populated cells $k$, not the schema width $d$. Concretely, for a record with present set $S$ and a visible subset $V \subset S$, we minimize

$$ \mathcal{L} = \sum_{j=1}^{d} \mathrm{BCE}\big(\mathbb{1}[j \in S],\ \sigma(e_j^\top h_V)\big) + \lambda \sum_{j \in S \setminus V} \ell\big(x_j,\ f(h_V, e_j)\big). $$

Here $h_V$ is the pooled encoder output on the visible pairs, $e_j$ is the learned embedding of attribute $j$, and $\sigma$ is the logistic function, so the first term is a cheap dot-product head that predicts the full bitmap. In the second term, $x_j$ is the held-out value, $f$ is a value decoder queried by attribute $j$, $\ell$ is a binned cross-entropy loss, and $\lambda$ trades off the two terms. The encoder's attention cost is $O(|V|^2)$. The presence head costs $O(d)$ per record. Neither ever puts a null into the sequence.

How This Differs From Prior Work

Several papers supply parts of the method without combining them. Interpreted storage stores only present pairs and omits nulls entirely; it was developed by Beckmann et al. and Baid et al.. LogKV separates the bitmap from the payload and reports 15:1 compression on logs. These supply our input format and loader design, but neither feeds a neural model.

Three tabular pre-training papers are the closest competitors, and each makes a choice we reverse:

  • TabSTAR turns rows into text for a partly unfrozen text encoder. Its sequences therefore grow with schema width.
  • TAPTAP does GPT-2 generative pre-training over serialized rows and is limited to tables with fewer than 100 columns.
  • TabRet uses masked autoencoding with shuffle augmentation, which deliberately destroys row-level structure. It also treats missing entries with masking heuristics.

None of the three treats which fields are present as a prediction target.

Two health-record models already feed only present items, a design closer to ours. MedM-PLM represents codes as sets of positive codes and predicts masked codes. The numeric-encoding study above also omits absent tests. These two constrain the proposal: they show that "just drop the nulls" is already standard in health records. Our contribution is therefore not dropping nulls. It is three things:

  • making the full-width bitmap an explicit pre-training target;
  • testing whether that target matters under informative versus random missingness;
  • carrying the design to non-health-record wide tables such as surveys and logs.

FTT+ restricts attention to feature-to-summary-token links during pre-training. That is sparsity in the attention pattern, not in the data, and its evaluations use dense tables. We reuse its summary-token bottleneck as an optional encoder variant.

On the evaluation side, RADAR motivates the proposal. It shows frontier models dropping by up to 59% on tables with missing data and other perturbations. It also shows direct prompting collapsing toward zero as tables approach 16K tokens. Finally, Speedrunning Tabular Foundation Model Pretraining supplies a fast pre-training recipe, but its synthetic generator produces dense tables with at most 20 features. We extend that generator to produce controlled missingness.

Research Program

The work has three parts.

Interpreted-record encoder and loader. Each present pair becomes one token: an attribute embedding plus a value embedding. Numeric values use combined binning, the stable default recommended by the numeric-encoding study, with bin count set by its fitted law $B(N) \approx 1.14N^{0.237}$. Categorical values use per-attribute embeddings. Free text uses a small text encoder, as in TabSTAR. The loader reads bitmap-stripped columnar files directly, building token sequences from the bit vector and the dense payload without ever materializing wide rows. We will measure the throughput gain of this loader. That gain is an engineering byproduct, not the scientific claim.

The factorized objective. We pre-train with the presence-plus-value loss above. In the comparisons, we vary only the objective, holding encoder, data and compute fixed.

A controlled missingness testbed. We modify the synthetic prior of modded-nanoTabPFN to emit wide tables with 200 to 2,000 attributes and 1% to 10% fill. Missingness is generated under three regimes:

  • missing completely at random;
  • missing at random given observed columns;
  • missing not at random, where presence depends on the label or on the unobserved value itself, as when a lab test is ordered because a disease is suspected.

The testbed lets us state in advance where the presence objective should help (the informative regimes) and where it should not (the fully random regime).

Baselines

Gradient-boosted trees. The naive but strong reference point is XGBoost, LightGBM and CatBoost trained directly on the downstream table. These models route nulls natively, and we also give them explicit missingness-indicator columns. Each gets the same hyperparameter-search budget as our fine-tuning stage. On BRFSS→PKIHD, TabRet reports a test AUC of 79.74 for XGBoost and 76.50 for CatBoost.

TabRet. This is the strongest existing method for transfer across mismatched schemas in the target regime. It reports a test AUC of 82.70 on PKIHD after pre-training on the Behavioral Risk Factor Surveillance System (BRFSS) survey, which has 2.03M rows and 74 columns. That is +3.87 points over training on downstream data only. We rerun TabRet with its published pipeline on our exact splits. Beating it means exceeding its rerun AUC on PKIHD and on the other three BRFSS tasks (Diabetes, HDHI, Stroke), averaged over five seeds, with non-overlapping confidence intervals.

Clinical baseline. For MIMIC-III, the reference is MedM-PLM, which reports a 30-day readmission AUC of 74.70% against 69.37% for ClinicalBERT. We use that number as the target. Our model will use the structured data together with a lab and chart-event table spanning thousands of item IDs, a table MedM-PLM does not model.

Verbalized-text baseline. This is a TabSTAR-style model that turns rows into text with explicit null tokens. It shows the accuracy and cost of paying for width.

Isolating ablations. The ablation that isolates our key ingredient is the identical present-only encoder pre-trained with masked value prediction alone ($\lambda$-only, no presence head). A second ablation feeds the full $d$-bit bitmap as dense input with no presence objective. All pre-trained variants get equal GPU-hours and the same pre-training corpus.

Why not just…?

Why not just use gradient-boosted trees with missingness indicators? Trees see the null pattern directly and are very hard to beat with abundant labels. We expect them to tie or win in the full-data regime, and we will report that honestly. The evidence that pre-training helps is in the low-data and new-schema settings. TabRet beats XGBoost by about 3 AUC points with roughly 50% column overlap. FTT+ loses only 0.03 accuracy when downstream data shrinks from 800 to 200 samples. Our learning curves at 1%, 10% and 100% of labels will show where, if anywhere, the advantage lies.

Why not just drop null tokens and pre-train normally? This is the real threat, because a model that sees which pairs are present already sees absence implicitly. If the masked-value-only ablation matches the full objective under the informative regimes of the synthetic testbed, the presence head is unnecessary. In that case the contribution shrinks to an efficiency result, and we will say so. Our hypothesis is narrower. Masking hides presence and value together, so a value-only loss never rewards the model for anticipating which tests or fields get recorded. That anticipation is the signal the numeric-encoding study found to be nearly as predictive as the values themselves.

Why not just add a dense missingness-indicator vector as input? This doubles input width. It brings back exactly the $O(d)$-per-token cost that interpreted storage removes, and quadratic attention cost if each indicator becomes a token. The dense-bitmap ablation measures both its accuracy and its compute. If it matches us on accuracy at acceptable cost for $d$ in the hundreds, we will report the width at which it stops being practical.

Why not just prompt a long-context model with the serialized table? RADAR shows direct prompting collapsing toward zero near 16K tokens and dropping up to 59% under missing data and other perturbations. It also shows that code agents fall back on discarding rows with missing values. We will run the RADAR missing-data subset as a stress check, but we do not expect prompting to compete on wide records.

Proposed Experiments and Decisive Checks

The first decisive check is the synthetic interaction. On the extended nanoTabPFN prior, we compare the full objective against the masked-value-only ablation. If the presence head is doing what we claim, its gain should be near zero when data are missing completely at random and should grow with how strongly presence depends on the label or the hidden value. If the gain appears even under random missingness, it is a regularization effect, not missingness modeling, and we will say so.

The second check is real-domain transfer:

  • BRFSS→{Diabetes, HDHI, PKIHD, Stroke}, where survey skip patterns make missingness structured;
  • MIMIC-III readmission and medication recommendation using a wide lab and chart-event table;
  • a TabArena subset where we deliberately do not subsample features, to keep width.

The metric is AUROC at each label fraction, against TabRet, MedM-PLM's reported figures, tuned trees and the verbalized baseline.

The third check is cost. We report tokens per record, pre-training FLOPs and loader throughput as width $d$ grows at fixed fill, against the verbalized and dense-bitmap baselines. If our cost tracks $k$ rather than $d$, this confirms the scaling argument.

The proposal succeeds only if all three hold:

  • the presence head beats the value-only ablation under informative missingness;
  • it beats rerun TabRet on BRFSS transfer;
  • it does both at cost that scales with populated cells.