Teaching Small Models to Read Nested Table Headers Without Writing Them Out

Real tables in statistical reports, airline filings and financial statements rarely have a single row of column names. They have nested headers: a column labeled "2019" sits under "Female" under "Employed", and a row labeled "Ontario" sits under "Canada". Language models read such a table as one long flattened string, and in that string they lose track of which headers a number belongs to. The best current fix has a large model first rebuild the header trees and then rewrite every cell with its full lineage ("Canada > Ontario" + "Employed > Female > 2019" → 412). This works well for 70B-scale models, but it inflates the prompt and has never been shown to work for small, locally deployable ones.

We propose to teach a small model to do this header binding inside its hidden states rather than in text, and to keep the arithmetic explicit and executed. A large teacher builds the header trees and writes a verified program for each question. The student is then trained on two kinds of targets. At a middle layer, each cell's representation must point to its row and column header ancestors. At a deeper layer, the representation must flag which cells the teacher's program uses. At the output, the student writes the program itself, which a Python interpreter runs. At test time the student sees only the compact raw table and never writes the lineage paths. The bet is that structural parsing can be absorbed into the model's layers, while arithmetic cannot and should stay in executed code.

Core Claim

A 1–3B student distilled this way should approach large-model accuracy on hierarchical-table question answering while using a small fraction of the tokens needed by explicit lineage prompting. Its advantage over explicit-trace distillation should grow with header depth and table size.

This claim takes a position on two tensions in the literature. The first is about arithmetic. Latent Chain-of-Thought Improves Structured-Data Transformers argues that reasoning over structured data can happen entirely in continuous hidden states: it trains recurrent transformers from scratch on each dataset and reports gains on tabular and time-series benchmarks. Chain-of-Thought Reasoning in Tabular Language Models points the other way, showing that removing the external calculator from its 0.8B pipeline drops TabMWP accuracy from 92.15% to 74.58%. We read these as compatible. Locating and binding cells is a pattern-recognition job that can be done in hidden states, while computing with the retrieved numbers is not, so we move only the first into the layers.

The second tension is about how to present a table. TableMaster finds that verbalizing a table as prose helps, and it needs external conversion to handle HiTab. Orthogonal Hierarchical Decomposition (OHD) finds that explicit tree paths prevent "structural collapse." We keep OHD's structural signal but deliver it as supervision during training rather than as prompt text at inference.

Relation to Prior Work

Three papers motivate the gap. HiTab introduces the hierarchical-table benchmark and shows flat-table parsers degrade with depth, reaching about 14.1% at depth five or more. OHD shows DeepSeek-v3 scores only 15.85% exact match (EM) on raw HiTab tables but 67.74% with tree induction and lineage paths. OHD also notes the token overhead of those paths and tests nothing below 72B.

Four papers show that table reasoning distills into small models, but each leaves structure implicit in text. Effective Distillation of Table-based Reasoning Ability from LLMs fine-tunes Flan-T5-base on filtered teacher rationales. It reaches 78.72% TAPAS-Acc on SciGen, above gpt-3.5-turbo's 72.34%, but only on flat tables and only through text-level sequence training. DARGO never updates student weights and instead retrieves teacher-refined plans at test time. PoTable and Chain-of-Table define staged table workflows but run them only on large models.

The method comes from the chain-of-thought distillation line. Implicit Chain of Thought Reasoning via Knowledge Distillation moves reasoning from generated tokens into layer depth by matching a student to a teacher's hidden states. We reuse the idea of placing supervision at intermediate layers. We differ in three ways: our targets are discrete structural labels rather than teacher hidden states, which avoids the projection layers needed between mismatched architectures; we apply it to tables; and we internalize only part of the reasoning.

That paper also constrains the design. Its implicit variant reached only 22% on GSM8K-Aug versus 44% for explicit chain-of-thought, and a 12-layer GPT-2 failed where a 24-layer one succeeded. Internalizing arithmetic is therefore a bad bet, and student depth is a variable we must sweep. KPOD and Skip-Thinking show that uniform token-level training of small models blurs the tokens that matter, which is further reason to supervise binding directly instead of hoping it emerges from long traces. The vision-distillation work summarized in the file supports the general principle that intermediate-stage supervision beats output-only supervision. The limitation it flags, that rigid layer-to-layer matching over-constrains small students, is why we attach light auxiliary heads rather than force whole-layer alignment.

Research Program

Teacher labeling. For each HiTab training table, a large open model runs OHD-style tree induction to produce row and column header trees. Where HiTab's own hierarchy annotations exist, we check the induced trees against them. The teacher then writes a pandas program over lineage-indexed cells. We keep only programs whose execution matches the gold answer, following the verify-then-keep logic of the SciGen distillation work. From each kept example we derive three targets. The first is each cell's header ancestors, one per header level. The second is the set of cells the program reads. The third is the program text.

Student training. The student is a 1–3B decoder that reads a compact grid serialization of the table. Here $h^{(\ell)}_c$ denotes the hidden state of cell $c$ at layer $\ell$, $a_c$ is its set of true header ancestors, $r_c \in \{0,1\}$ marks whether the teacher's program uses $c$, and $p$ is the program. The training loss is

$$\mathcal{L} = -\log P(p \mid \text{table}, q) + \lambda_1 \sum_{c}\sum_{a \in a_c} \mathrm{CE}\big(\mathrm{ptr}(h^{(\ell_1)}_c),\, a\big) + \lambda_2 \sum_c \mathrm{BCE}\big(g(h^{(\ell_2)}_c),\, r_c\big),$$

where $q$ is the question, $\mathrm{ptr}$ is a pointer head scoring the header-token positions of the same table, $g$ is a linear relevance head, and $\ell_1 < \ell_2$ are a middle and a deeper layer. The ordering is deliberate: binding comes first and selection second, mirroring how the stages depend on each other.

Inference. The heads are discarded. The student emits only a short program, which is executed by an interpreter. The main analysis sweeps student size, the placement of $\ell_1$ and $\ell_2$, and header depth.

Domain regime. Hierarchical tables are the natural format of statistical agencies, airlines and financial filings. We therefore test whether header binding learned on HiTab transfers zero-shot to AITQA's airline tables and to FinQA's financial tables. This treats the domain gap as a transfer question about structure. It does not address missing domain formulas: TheoremQA shows those are not fixed by supplying definitions, and we do not claim to fix them either.

Baselines

Naive reference. The same student is fine-tuned on question-to-answer pairs over flattened tables, with no program and no auxiliary heads. Its reference points on Lacuna are the HiTab raw-table result for DeepSeek-v3 (15.85% EM) and the flat-parser MAPO baseline in HiTab (about 29% execution accuracy).

Strongest existing method. OHD prompting a large model. It reports 67.74% EM on HiTab with DeepSeek-v3 and 69.34 EM on AITQA with Qwen2-72B, Orthogonal Hierarchical Decomposition. We rerun OHD with our open teacher to get a same-teacher ceiling and record its prompt tokens per question. We do not expect a small student to beat this ceiling. Success means recovering at least 85% of the teacher-OHD EM at under one quarter of its tokens.

Strongest small-model competitor. Explicit-trace distillation, in the spirit of Effective Distillation of Table-based Reasoning Ability from LLMs and extended to hierarchy. The same student is fine-tuned on the full teacher output: lineage-annotated cells, then the program. It uses the same kept examples, the same compute and the same hyperparameter search budget. Beating it means matching its EM overall while using far fewer inference tokens, and beating its EM on the depth-four-or-more and largest-table strata, where explicit lineage crowds the context.

Rule-based lineage serializer. Headers are expanded deterministically from HiTab's merge metadata and prepended to each cell, then the student is fine-tuned on programs. This tests whether a teacher is needed at all.

Output-only program distillation. The same kept programs are used without auxiliary heads. This is analogous to what DARGO gets from teacher plans, except with weight updates.

Ablations. Two ablations isolate the key ingredient. The first moves both heads to the final layer, testing whether depth placement matters or whether any multi-task signal helps. The second drops only the lineage head. A no-teacher latent-recurrence student in the style of Latent Chain-of-Thought, adapted to emit programs, tests whether extra compute without structural targets is enough.

Why not just…?

Why not just distill the full explicit OHD trace into the small model? This is the real threat, and on small, shallow tables we expect it to tie. The SciGen result shows explicit rationale distillation works well at 220M on flat tables. Our claim is narrower: explicit lineage costs tokens that grow with the number of cells times header depth, and OHD itself reports degradation on larger tables (a 7.09-point drop from small to large). We report accuracy and tokens stratified by depth and size. If explicit distillation matches us in every stratum at an acceptable token cost, the proposal's accuracy claim fails and only an efficiency claim remains. We will say so.

Why not just expand headers with deterministic rules and skip the teacher? When clean merge markup exists, this may do nearly as well for binding, and the rule-based baseline measures that directly. It still pays the token cost at inference unless combined with our internalization. It also fails on tables whose hierarchy is implied by indentation or wording rather than markup, which is why OHD uses a large model to judge semantic header relations. We report results separately on HiTab tables with and without reliable merge metadata. We also test a variant in which rule-derived labels, rather than teacher labels, train the auxiliary heads. If that variant matches the teacher-labeled version, the teacher is unnecessary for the lineage targets.

Why not just prompt a frontier model with OHD, or scale the student up? Accuracy is not the issue: OHD with large models is our ceiling. Regulated financial and operational tables often cannot leave the premises, and the file's summaries of TaCo and DARGO cite exactly this deployment constraint. On scaling, Latent Chain-of-Thought found a twice-as-deep model overfit small structured datasets. We still sweep student size so that "a bigger student without heads" is measured, not assumed.

Proposed Experiments

The primary experiment trains on HiTab and reports EM and execution accuracy on the HiTab test set, stratified by header depth and table size, alongside input-plus-output tokens and wall-clock latency per question. The transfer experiment evaluates the HiTab-trained students zero-shot on AITQA (EM) and on FinQA (execution accuracy, against FinQANet's reported 61.24%, FinQA). It asks whether header binding carries across domains.

The first decisive check is mechanistic. On held-out tables, we probe whether lineage-pointer accuracy at $\ell_1$ predicts answer correctness. We also intervene: we corrupt the student's middle-layer representation of a cell by swapping in a representation from a cell with different headers, and test whether the program changes to reference the wrong cell. If binding accuracy is high but unrelated to answers, the auxiliary head is decorative.

The second decisive check is the depth ablation against final-layer heads. If placing supervision at intermediate layers gives no gain over the same targets applied at the last layer, the hierarchical part of the claim is falsified. The contribution would then reduce to multi-task distillation.

The third check repeats the student-size sweep from the smallest to the largest student. Given the depth failure reported for implicit chain-of-thought, we expect a minimum size below which internalized binding breaks down. Locating that threshold is itself a useful result for anyone deploying small table readers.