Teaching a Tabular Model Where to Look with LLM Rationales

When a table has only a few dozen labeled rows, a gradient-boosted tree has no way to know that "Systolic BP" and "Age" probably matter for a cardiac outcome. A large language model (LLM) does know this, because it has read about blood pressure and aging. Today the main way to use that knowledge is to turn every row into a sentence and run a multi-billion-parameter model on it, which is too slow for high-volume tables. We propose to spend the LLM only at training time and only on the few labeled rows. For each labeled row, the LLM writes a short explanation of its label. We parse which columns the explanation cites, and use those citations to supervise the part of a small, linear-time tabular model that decides how much each column should influence its internal state. At deployment the LLM is gone, and the small model has learned from the LLM where to look.

The small model is a selective state space model (SSM), a recurrent network that reads a row's columns one at a time and computes, for each column, an input-dependent step size that controls how strongly that column is written into its running memory. Mambular: A Sequential Model for Tabular Deep Learning showed that this kind of model is the strongest neural tabular learner on its benchmarks and runs in time linear in the number of columns. That per-column step size is a natural, per-row "attention budget" that we can supervise directly. Our claim is that aligning this budget with the columns an LLM cites closes much of the few-shot gap between cheap tabular models and LLM-based classifiers, at a fraction of their inference cost. We also claim that the gain comes from the LLM's semantic prior, and not merely from having any feature-importance signal. The second claim is the one a skeptic will attack, so the experiments are built around it.

Core Claim

Explanations help small fine-tuned models but can hurt large ones. Can LLM-Generated Textual Explanations Enhance Model Classification Performance? An Empirical Study found that GPT-4o-mini explanations raised BERT-base Macro F1 on e-SNLI from about 0.65 to 0.85, while adding explanations to zero-shot LLM prompts lowered accuracy by 7% to 15%. This paper motivates the proposal: rationales are most useful as a training signal for a small student. It also constrains the design. The same study found that rationales which merely restate the reasoning helped less than those which add outside knowledge. In a table, the outside knowledge an LLM can supply is mostly which attributes plausibly bear on the label. Exact numeric thresholds are where tokenized LLMs are weak and trees are strong.

We therefore extract only the column-relevance content of a rationale and discard its arithmetic. We then inject that content into a model whose numeric handling stays specialized. Mambular encodes each number with periodic linear encodings, meaning each value is mapped to a set of bins with a linear position inside its bin.

The objective has two parts. Let $g_{ij}$ be the channel-averaged selective step size the model assigns to column $j$ of row $i$. Let $p_i = \mathrm{softmax}(g_{i\cdot})$ be the resulting distribution over columns. Let $r_i$ be the normalized distribution of columns cited in the LLM rationale for row $i$, smoothed with a small uniform mass. We train with

$$\mathcal{L} = \sum_i \mathrm{CE}(y_i, \hat y_i) + \lambda \sum_{i \in \mathcal{R}} \mathrm{KL}(r_i \,\|\, p_i),$$

where $\mathrm{CE}$ is cross-entropy on the label, $\mathcal{R}$ is the set of rows that have rationales, and $\lambda$ sets the weight of the alignment term. The supervision is per row, not per dataset. This distinguishes it from global feature weighting: a rationale for a young patient and one for an elderly patient may cite different columns.

Research Program

The student model is a semantic variant of Mambular. Each column token adds a frozen text embedding of the column name to Mambular's value encoding. We add the name embedding because Mambular indexes columns anonymously. The name embedding does two jobs. First, it lets the model share structure across columns with related meanings. Second, it gives a principled column order: we order columns by clustering their name embeddings, so that semantically related columns sit near each other in the scan. This targets the order-sensitivity problem Mambular itself reports, where interacting columns placed far apart are attenuated. Text-valued cells are embedded with the same frozen encoder. We follow the method-supplying idea in CARTE: Pretraining and Transfer for Tabular Learning, which scales numeric values by column-name embeddings and runs a graph attention encoder, but we drop the graph attention encoder in favor of the linear scan.

Rationales come from an open model, Qwen3-8B, one of the models LLM Meeting Decision Trees on Tabular Data used for rule refinement. Each labeled row is serialized with the plain "The [column] is [value]" template, which TabLLM found to beat fancier LLM rewrites. We then ask the LLM to explain the given gold label and to name the columns it relies on. Providing the gold label avoids rationalizing wrong answers. It does risk plausible but unfaithful justifications, so we compare against a variant in which the LLM predicts first and we keep only rationales whose prediction agrees with the label. Citations are mapped to columns by exact name matching, with a fallback of embedding similarity between the rationale and the column names.

The total LLM cost is one generation per labeled row, which means tens to hundreds of calls per dataset. Deployment cost is a single SSM forward pass per row. We report both costs explicitly against per-row LLM inference.

This differs from its neighbors as follows. Relative to DeLTa, which queries an LLM once per dataset to refine random-forest rules and fits small gradient networks per partition, our supervision is per-row and acts on a neural model's internal gating rather than on tree splits. Relative to Enhancing Tabular Learners with Context-Aware Semantic Embeddings (CASE), which encodes rows with a primed TabGemma cache and hands PCA-compressed embeddings to TabICL, we need no LLM at inference time and use no rationale-free embedding. Relative to the explanation study, we move rationale distillation from text classification into tables and from input augmentation into representation supervision.

Baselines

The naive reference points are CatBoost and XGBoost on raw features, tuned with the same random-search budget as every other method on each shot count. CatBoost had the best average rank in the Mambular paper, so these are serious baselines. Beating them at 16 to 64 shots is required. Matching them at 256 or more shots is acceptable, since TabLLM reports that trees catch up in that regime.

The key ablation isolates the rationale term. We run anonymous Mambular, and semantic Mambular trained with the identical architecture, data, and budget but with $\lambda = 0$. The proposal succeeds only if the rationale-aligned model beats semantic Mambular with $\lambda = 0$, averaged over seeds and datasets. A gain that shows up only against anonymous Mambular would mean the column-name embeddings, not the rationales, are doing the work.

The strongest existing methods differ by regime. On text-rich tables, the strongest is CASE combined with TabICL. The file reports that it beats the next-best baselines such as AutoGluon by 2–3 percentage points on the CARTE and TextTab benchmarks; no per-dataset number was found on Lacuna, so we rerun it with its released configuration on our splits. On few-shot tables with meaningful column names, the strongest is TabLLM, which reports 0.84 AUC zero-shot on Income and more than 5 AUC points over XGBoost and LightGBM in the 0–64-shot range. For CARTE itself, no reported per-dataset result was found on Lacuna, so it is also rerun. Beating CASE+TabICL or TabLLM outright is not required. The success criterion is to come within one AUC point of them at 16–64 shots while using at least two orders of magnitude fewer inference FLOPs.

Finally, we include a confidence-routed cascade in the style of Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems. That work routes rows with low distance-to-uniform confidence from small to large models and reports matching the large model's accuracy on QQP with 11.07% of its compute. Here the cascade is semantic Mambular sending uncertain rows to the rationale-prompted LLM. The cascade is scored on the same accuracy-versus-FLOPs curve as our model, which never calls the LLM at test time.

Why not just…?

Why not just supervise the gates with tree feature importances instead of an LLM? This is the most serious threat. SHAP values from a CatBoost model fit on the same labeled rows are free and per-row. If they work as well as rationales, the LLM contributes nothing. Our bet is that at 16–64 rows the tree's importances are fitted to noise, while the LLM's citations reflect a prior learned from text. We test this directly with an arm that uses SHAP-derived targets $r_i$. We also include a shuffled-rationale arm that keeps the citation counts but permutes which row gets which rationale, and a uniform-target arm. If SHAP matches the LLM at every shot count, we will report that the method reduces to importance-guided gating and drop the LLM from the recommendation.

Why not just feed the rationale text in as extra features? Embedding each row's explanation as an input, as in the explanation study's small-model setting, requires generating a rationale for every test row. That puts the autoregressive LLM back on the serving path, which is exactly the cost this proposal removes. We include it as an upper-bound arm, using per-row rationale embeddings at test time. The gap between that arm and our model measures how much of the rationale's value survives distillation into the gates.

Why not just run a cascade and call the LLM on hard rows? A cascade is a sound design and appears among our baselines. It still pays for LLM calls at test time, and it has the threshold-calibration fragility noted for distance-to-uniform confidence under shift. The two ideas also compose: a better stage one sends fewer rows onward. We therefore report the escalation rate needed to reach a fixed AUC for a cascade built on each first stage, with and without rationale alignment.

Why not just use CatBoost and accept the few-shot gap? On purely numeric tables this may be the right answer, and CASE itself concedes an edge to statistical learners on TabArena. We use TabArena as a negative control. There we predict no gain from rationales, because column names carry little signal. A gain there would suggest the alignment term is acting as a generic regularizer rather than a semantic channel.

Proposed Experiments

We evaluate at 16, 32, 64, 256, and full-data shot counts, using ten seeds with shared splits. The datasets are the public TabLLM datasets (Income, Bank, California, and the others), the CARTE benchmark classification tasks, TextTab, and the six DeLTa classification sets (Blood, Credit, Car, Bank, Adult, Jannis) through the released DeLTa codebase data format. TabArena serves as the negative control.

The metrics are AUC and accuracy. Cost is measured three ways: inference FLOPs, latency per 1,000 rows, and one-time LLM calls. We also measure faithfulness of the learned gates, defined as the drop in AUC when the top-$k$ gated columns of a test row are masked, compared with masking $k$ random columns.

Two checks are decisive. The first is whether rationale alignment beats both the $\lambda = 0$ and SHAP-target arms at 16–64 shots on the TabLLM and CARTE datasets, and shows no gain on TabArena. The second is whether the column-order ablation (random order versus semantic clustering) interacts with alignment, which would tell us whether rationales partly repair Mambular's order sensitivity. A secondary study varies rationale quality: gold-conditioned rationales, agreement-filtered rationales, and rationales from a smaller model. This asks whether the plausibility trap in synthetic explanations translates into misdirected gates on tables.