Target-Aware Readouts of Frozen Encoders for Multimodal Tables

Many prediction tasks pair a table with a photo or a passage of text: a chest X-ray with patient demographics, a wine review with price and region, an in-game item image with its market history. The standard recipe passes the image or text through a frozen pretrained encoder (a network trained on web-scale data and then left unchanged), appends the resulting embedding vector to the table, and trains a gradient-boosted tree model on the combined columns. MulTaBench showed that this recipe loses a lot. Fine-tuning the last encoder layers on each dataset's own target (their Target-Aware Representations, or TAR) beat frozen embeddings on all 40 benchmark datasets. The cost is a per-dataset gradient-training loop that can make text preprocessing up to 10x slower. That loop also gives up the property that makes in-context tabular models like TabPFN attractive: they need no training on the user's data.

This proposal asks where the loss actually happens. One possibility is that the frozen encoder throws the information away. The other is that the information is still in the vector, but tree models read it poorly. Tree models split on one coordinate at a time, so a signal spread across hundreds of dense coordinates, or one that matters only for certain rows of the table, is hard for them to find. If the readout is the problem, then cheap closed-form features computed from the frozen embedding should recover most of TAR's gain without touching the encoder. These features would be fit to the residual errors of a table-only model, and some would depend on the table's own columns. We propose a residual-driven feature search that builds such features, and an experiment that splits TAR's gain into "encoder" and "readout" parts.

Core Claim

Most of the gap between frozen and fine-tuned unstructured representations on multimodal tabular benchmarks can be closed in three steps, all without updating any encoder weight.

First, fit an ordinary model on the table alone and compute its errors (residuals) on held-out folds.

Second, generate a small set of candidate features from the frozen embedding that target those residuals. These come in three families:

  • linear probes from the embedding to the residual;
  • probes fit separately within groups defined by table columns, or onto embedding-times-column products, which capture interactions that cross the modality boundary;
  • similarity scores to short concept phrases that a language model proposes after inspecting the worst-predicted examples.

Third, keep only the candidates that reduce residual error, scored cheaply in the style of OpenFE.

The output is a short list of named, inspectable scalar features that any tabular learner, including an in-context one, can use.

The concept-similarity family has one piece of geometry that matters. In a contrastive image–text model such as CLIP, image and text embeddings sit in two separated cones. Mind the Gap showed that this separation comes from architecture and the contrastive loss, and that it shifts downstream accuracy. A raw cosine between an image and a text concept therefore mixes the concept's meaning with a constant offset between the cones. We define the concept feature after removing each modality's mean:

$$s_c(x) = \cos\big(f_{\text{img}}(x) - \mu_{\text{img}},\; f_{\text{txt}}(c) - \mu_{\text{txt}}\big),$$

Here $f_{\text{img}}(x)$ is the frozen image embedding of example $x$, $f_{\text{txt}}(c)$ is the text embedding of concept phrase $c$, and $\mu_{\text{img}}, \mu_{\text{txt}}$ are the mean embeddings of each modality computed on training data. For text-tabular datasets, the encoder (e5) embeds both data and concepts in one space, so no correction is needed. Those datasets act as a natural control for whether gap correction matters.

Relation to Prior Work

Two strands motivate the work.

MulTaBench supplies the evidence that frozen embeddings underperform, and it names gradient-free target-awareness as an open problem. We take its benchmark and its TAR method as the main competitor. We differ in never fine-tuning the encoder.

Mind the Gap shows that the gap between modalities is structural and should be handled deliberately rather than ignored. We use its finding only in the narrow form of mean-centering before cross-modal similarity, which can be tested as an ablation.

Three strands supply the method.

OpenFE scores candidate features by fitting light models to a base model's residuals (FeatureBoost) and prunes candidates with successive halving. It reports, for example, beating 99.3% of teams on IEEE-CIS Fraud Detection. OpenFE's candidates are arithmetic combinations of scalar table columns. Our candidates are projections of dense embeddings, sometimes conditioned on table columns. OpenFE's operators cannot form these.

CAAFE has a language model write feature code from column descriptions. It raised mean ROC AUC from 0.798 to 0.822 on its 14-dataset suite. Demirel et al. has a language model read about 40 representative patient records and write a rubric that becomes deterministic extraction code. Their Global-Rubric-Tabular variant reached mean AUROC 0.770 on EHRSHOT using records of text and billing codes only. We borrow the idea of a language model inspecting a few chosen examples. We aim it at the highest-residual examples, though, and its output is concept phrases scored in embedding space rather than parsing code.

One strand constrains the design. Küken et al. found that language-model feature generators fall back on simple arithmetic. This is why we do not let the language model propose features over table columns at all. Its only job is naming visual or textual concepts. Residual probes and the scoring step do the rest.

Two end-to-end alternatives exist but solve a different problem. TIP pretrains a joint image–table transformer from scratch. FindRec aligns embedding distributions inside a recommender. Both produce dense hidden vectors and require joint training, and neither offers a drop-in feature set for frozen tabular learners.

Research Program

All candidate features are built by nested cross-fitting. Probes and concept selections are fit on inner folds and applied to the held-out outer fold. This guards against target leakage, the main way a residual-fitted feature can look good spuriously.

Base model. The table-only model is CatBoost with default settings.

Residual probes. These are ridge regressions (logistic regressions for classification residuals) from the embedding, with the penalty chosen by inner cross-validation.

Cross-modal conditional probes. These come in two forms:

  • For each categorical column with a manageable number of levels, fit a separate probe per level, with shrinkage toward the global probe.
  • For each numeric column, fit a probe on the elementwise product of the embedding and the standardized column.

Concept candidates. We caption the 30 highest-residual images with an off-the-shelf captioner, or show the raw text for text datasets. A language model then proposes up to 50 short contrastive phrases describing what distinguishes over-predicted from under-predicted cases. Each phrase becomes a score $s_c$.

Selection. All candidates enter FeatureBoost-style scoring with successive halving. At most 20 features survive.

Analysis. The analysis has three parts.

The first is the gain decomposition. For each dataset we measure four quantities:

  • frozen-concatenation performance;
  • the readout gain, that is, the improvement our features add on top of the frozen embedding;
  • TAR's gain;
  • a residual measure: what TAR still adds when its embedding is appended to our features.

If the readout hypothesis is right, our features should account for most of TAR's gain, and the residual measure should be small.

The second part checks whether the remaining encoder gain is predictable from dataset properties. We expect it to concentrate where the task depends on fine local detail that a global embedding plausibly discards, such as texture defects on CS:GO skins or small findings on CheXpert. We expect it to be smaller where the signal is semantic, such as wine-review sentiment or toxicity.

The third part reports wall-clock cost per dataset against TAR's.

Baselines

The naive reference is frozen concatenation: the raw frozen embedding (e5-small for text, DINO-v3-small for images, matching MulTaBench) appended to the table and passed to CatBoost, LightGBM and TabPFN. A second naive point compresses the embedding to 32 principal components before appending, since tree models often do better with fewer dense columns. Our method must beat both on mean AUC or R² across the 40 MulTaBench datasets, with a sign test over datasets.

The strongest competitor is TAR from MulTaBench: LoRA fine-tuning of the last three encoder layers on the target, with regression targets binned into 20 classes. It is the only method shown to fix the frozen-embedding loss on this benchmark. Its reported gains over frozen embeddings are a mean of +0.022 (AUC or R²) on image-tabular datasets and +0.018 on text-tabular datasets, about a 90% win rate for CatBoost and LightGBM, and +0.119 on Jigsaw Toxicity. No reported per-dataset absolute scores were found on Lacuna for TAR, so we rerun it ourselves with the same encoders, splits, downstream learners and hyperparameter budget, and verify that our reproduction lands near those mean gains before comparing. There are two success thresholds. Recovering at least three quarters of TAR's mean gain at under a fifth of its preprocessing time would make our method a practical replacement. Matching or exceeding TAR, or adding significantly on top of it, would be a stronger result.

A second strong competitor runs OpenFE on the table columns, uses frozen embeddings for the unstructured input, and trains separately. This tests whether any gain we report is just better table features.

On text-tabular datasets, CAAFE run on the table with the text field described in its prompt tests whether language-model code generation alone suffices.

The ablations isolate the key ingredient:

  • a plain out-of-fold probe from the embedding to the target, rather than to the residual;
  • residual probes without the conditional family;
  • the full method without concept features;
  • concept features without gap correction.

Why not just…?

Why not just fine-tune the encoder, as TAR does? On raw accuracy TAR may simply win, and the decomposition will say so. Fine-tuning has two concrete costs on the evidence in the file. Text TAR preprocessing runs up to 10x longer than frozen preprocessing, and the per-dataset gradient loop breaks the no-training workflow that Block Redundancy in Tabular Foundation Models for Clinical Applications identifies as the reason clinical sites use in-context tabular models. If our features recover most of the gain, these costs buy little. If they do not, we will have measured exactly which datasets need the encoder changed.

Why not just stack a single out-of-fold probe that predicts the target from the embedding? This is the real threat, because it is target-aware, gradient-free and nearly free to compute. It is included as an ablation, not argued away. Our method should beat it in two situations. One is when the useful signal lies in what the table cannot already explain, which a target-fitted probe partly spends re-learning. The other is when the image or text means different things for different table groups, which a single global direction cannot express. If the plain probe matches the full method across datasets, we will report that as the main finding: the frozen-embedding loss is a readout problem with a one-line fix. The added machinery would then not be justified.

Why not just serialize everything to text and let a language model write features, as CAAFE does? CAAFE never sees images. On text it inherits the simplicity bias that Küken et al. documented, and wide prompts invite references to columns that do not exist. The CAAFE baseline on the 20 text-tabular datasets measures this directly.

Why not train an end-to-end image–table transformer like TIP? TIP's gains on cardiac imaging and car data came from pretraining on large paired cohorts. Most MulTaBench datasets are far smaller, and the result is a hidden vector that cannot be audited. We include TIP on the two image-tabular datasets where its training budget is affordable, as a reference rather than a full comparison.

Decisive Checks

The main experiment covers all 40 MulTaBench datasets with three downstream learners and five outer folds. It reports mean AUC or R², win rate against frozen concatenation and against TAR, and preprocessing wall-clock time.

The decisive result is the gain decomposition. If our features capture most of TAR's improvement and appending TAR's embedding on top adds little, the frozen-encoder bottleneck is mostly a readout bottleneck. If TAR keeps a large residual advantage, the encoder genuinely discards information. In that case the per-dataset pattern, whether local-detail image tasks or semantic text tasks, tells practitioners when fine-tuning is worth its cost.

Two smaller checks complete the picture. The gap-correction ablation on image datasets tests whether the cone geometry from Mind the Gap affects feature quality in practice. The TabPFN runs test whether the selected features make an in-context tabular model competitive on multimodal data without any training on the user's data.