When a language model reads a table, it reads a flattened string. The table goes row by row into Markdown or CSV, so each cell ends up far from the column header that gives it meaning. For English this gap is already a known source of errors. For Arabic, Tamil or Swahili it is worse in a way nobody has measured. The tokenizer of an English-centric model, the component that splits text into the units the model reads, breaks these scripts into many tiny pieces. Every cell and every header becomes several times longer, so the distance between a value and its header grows by the same factor. This proposal asks whether a measurable share of the gap in multilingual table question answering comes from this inflation and not from failed transfer of reasoning. If it does, a cheap fix should recover much of that share. The fix is to add target-language tokens whose starting vectors are built from the model's existing pieces, while leaving the model's table skills untouched.
The test is a controlled factorial study. We train table skills in English only and freeze them. We then cross two factors. One is whether the vocabulary has been adapted to the target language. The other is whether the table is serialized in a format that already keeps headers next to cells. If inflation is the culprit, vocabulary adaptation should help most when the serialization leaves headers far away, and help little when headers are already attached to each cell. We run this at 8B and 70B parameters. That tells us whether the effect fades at scale, as larger models are often reported to absorb such problems.
Core Claim
We claim that, for scripts the base tokenizer fragments heavily, part of the zero-shot cross-lingual drop in table question answering comes from header-to-cell token distance, and that vocabulary adaptation removes this part. "Zero-shot cross-lingual" here means task supervision only in English, evaluation directly in other languages. The new element is the predicted interaction. The benefit of adding target-language tokens should be large under row-by-row Markdown and small under serializations that place the header beside every cell. A plain "vocabulary adaptation helps" result would not tell us which mechanism is at work. The interaction can.
A secondary claim concerns a puzzling finding in M3TQA. That paper built a 97-language table benchmark and trained LLaMA-3.1-8B with supervised fine-tuning and reinforcement-learned reasoning traces. It found that the traces help high-resource language families but lose to plain supervised fine-tuning on low-resource ones. We test one candidate cause. Reasoning traces in fragmented scripts become very long, and they quote cell values that are themselves heavily split into pieces. If that is the cause, vocabulary adaptation should shrink the gap between reasoning traces and plain supervision for those languages.
Prior Work and Positioning
Three groups of papers shape this proposal, and each plays a different role.
The motivation comes from table work. Table Fact Verification with Structure-Aware Transformer argues that one-dimensional position forces attention to rebuild the grid. Beyond Linearization: Attributed Table Graphs introduces TABGR. TABGR rewrites tables as <row_id, column_header, cell_value> triples and reranks them with question-guided PageRank. It shows that ordinary linearized baselines lose up to 18.7% accuracy when rows or columns are shuffled, and that the lost-in-the-middle effect persists even at LLaMA-3.3-70B. HInT finds "structural layers" in a 3B model and injects hypergraph features into them. All of this work is English-only and keeps the tokenizer fixed. M3TQA is multilingual but uses plain linearization and the stock LLaMA vocabulary. None of these papers asks how fragmentation interacts with structure, which is the question here.
The method comes from vocabulary adaptation. In that line of work, new tokens get starting vectors built from the original subwords they split into. Exploring Design Choices for Building Language-Specific LLMs finds that averaging the subword vectors matches more elaborate schemes. It also finds that about 10k added tokens is a sweet spot, and that adding just 1k tokens cuts fertility (tokens per word) in non-Latin scripts by about 3x. VocADT learns linear combinations through an adapter while keeping the transformer body frozen. That frozen body is the property we need, because it lets English-trained table skills survive the vocabulary swap untouched. WECHSEL uses bilingual static embeddings to pick similar source subwords. All three were evaluated only on unstructured text and at 7B scale or below. We reuse their techniques without modification. The new part is applying them to structured inputs and measuring the effect at 70B.
The constraints come from negative results. VocADT reports that linear combinations fail for scripts with near-zero base coverage, such as Hangul in Mistral. An Empirical Study on Cross-lingual Vocabulary Adaptation reports that English-centric Mistral degrades under simple initialization unless FOCUS or CLP+ is used. CLP+ blends weight estimates from a smaller auxiliary model with borrowing from semantically similar tokens. The design-choices paper reports that heavily multilingual bases like Gemma-7B gain little. These results mean the predicted effect should depend on the script and on the base model. We treat that dependence as part of the hypothesis rather than a nuisance.
Research Program
The pipeline has three stages, and none of them uses labeled target-language data.
In the first stage we fine-tune LLaMA-3.1-8B and LLaMA-3.3-70B on the English-language portion of M3TQA-Instruct. We use LoRA, low-rank adapter matrices added to the transformer layers, and train two variants, one with plain supervised targets and one with reasoning traces. If the released English portion turns out to be small, we will add English WikiTQ and TabFact training data and report that change.
In the second stage we add about 10k target-language tokens per language. Their starting vectors come from one of four schemes: mean subword initialization, VocADT, FOCUS, or CLP+. We then train only the embedding-side parameters on unlabeled target-language text, with the body and the table-skill LoRA frozen. Freezing makes the comparison clean. Any change in table accuracy can be attributed to the input and output representation, not to new task learning.
In the third stage we evaluate on M3T-Bench. It covers four table types and four tasks: numerical computation, cell extraction, factual verification, and open-ended questions. Each table is presented under three serializations. Row-major Markdown keeps headers far from cells. A header-attached form repeats the column header inside each cell. TABGR triples go through its reranking step.
We choose languages to span fertility and script coverage. German is a low-fertility control. Arabic, Hindi, Tamil, and Swahili overlap with prior vocabulary-adaptation studies. Japanese is included, and Korean serves as the known failure case for linear initialization.
For every example we log two diagnostics: the token distance between each answer-relevant cell and its column header, and total input length. These let us fit accuracy as a function of distance within and across languages.
A synthetic control separates distance from token quality. We take English M3T-Bench tables and force their cells and headers to be tokenized at character level. This reproduces the target-language length inflation in a language the model knows well. If English accuracy under forced fragmentation falls along the same accuracy-versus-distance curve as Tamil or Arabic, distance is doing the work. If the target languages fall well below that curve, then poor token quality or failed reasoning transfer dominates, and the core claim is weakened.
Baselines
The naive reference point is the English-fine-tuned model with its stock vocabulary, reading the target-language table directly in row-major Markdown. This is the M3TQA setting restricted to English supervision. M3TQA reports that its trained 8B model reaches parity with or exceeds GPT-4o in most scenarios. It gives no per-language or per-task numbers, so no reported result is available on Lacuna for an M3T-Bench point estimate. We will therefore rerun this baseline ourselves on the same splits.
The strongest structural competitor is TABGR, with its triple serialization and question-guided reranking. The paper reports 78.8% accuracy on WikiTQ and 93.6% on TabFact, both in English. It has never been tested cross-lingually. We run TABGR on the same base models and the same English-trained LoRA, with identical decoding settings, and we tune its reranking budget on the English development data only. The proposal beats TABGR if adapted-vocabulary row-major Markdown matches or exceeds TABGR's target-language accuracy at a lower input token count. A second win condition is that combining adaptation with TABGR improves low-resource scripts by more than the seed-level variance.
The strongest vocabulary-side competitor depends on the base model. For English-centric bases, the cross-lingual vocabulary adaptation study suggests this is FOCUS or CLP+. We include mean, VocADT, FOCUS, and CLP+ under an identical token budget for embedding training.
Translate-test runs the table and question through machine translation into English, then answers with the English model. We use the same base model and the same compute.
Finally, an ablation isolates the key ingredient. We keep the adapted vocabulary but initialize the new vectors randomly, with the same training budget. If random initialization matches subword-based initialization on table tasks, then the subword starting point is irrelevant here and only shorter sequences matter.
Why not just…?
Why not just use the header-attached or TABGR serialization and skip vocabulary changes? This is the most serious threat. If headers sit next to cells, inflation no longer separates them, and the structural part of our story disappears. The factorial design addresses this directly. If adaptation adds nothing under TABGR, we will report that structure-preserving serialization is enough for distance effects. The remaining case for adaptation would then rest on cost and on generation length: TABGR's triples repeat headers, which multiplies the fragmentation cost. That cost case is weaker, and we will say so.
Why not just translate everything to English? Zero-shot Reading Comprehension by Cross-lingual Transfer found that a machine-translation pipeline underperformed direct transfer. M3TQA needed a six-step pipeline with cell correction and back-translation validation to translate tables faithfully. Both results suggest that off-the-shelf translation damages tables. Translate-test still runs as a full baseline, and we report its per-language accuracy and cost alongside ours.
Why not just use a heavily multilingual base model, or rely on 70B scale? The design-choices paper shows that multilingual bases gain little from vocabulary extension. We expect the interaction to shrink or vanish for them, and we will check this on one such model. On scale, TABGR shows that 70B models still lose cells in the middle of long contexts. Our 8B-versus-70B comparison measures whether the slope of accuracy against header-cell distance flattens with size. A flat slope at 70B would mean scale alone solves the distance problem, a negative result we would report.
Proposed Experiments and Decisive Checks
The primary measure is accuracy per task on M3T-Bench, by language, model size, serialization, and initialization scheme, with three seeds per configuration. Secondary measures are input and generated token counts, wall-clock inference time, and the accuracy drop when rows and columns are shuffled.
The first decisive check is the interaction itself. The gain from vocabulary adaptation under row-major Markdown must exceed the gain under header-attached serialization for high-fertility languages, and the difference should be near zero for German.
The second check is the forced-fragmentation curve described above. English tables tokenized at character level must lie on a shared accuracy-versus-distance curve with the target languages for distance to count as the explanation.
The third check concerns the reasoning-trace puzzle. On low-resource languages, the deficit of reasoning traces relative to plain supervision should shrink after adaptation. If it does not, the M3TQA disparity is not mainly a tokenization effect.
We also expect Korean, under mean or VocADT initialization, to show smaller gains than under CLP+. That would replicate VocADT's reported failure case in the structured setting.
Any of these checks can fail independently. Each failure narrows the explanation for why multilingual table reasoning lags English, which is useful even if the core claim does not hold.
References
- M3TQA: Massively Multilingual Multitask Table Question Answering
- Table Fact Verification with Structure-Aware Transformer
- Beyond Linearization: Attributed Table Graphs for Table Reasoning
- HInT: Hypergraph Infusion at the Structural Layers Improves Table Understanding
- Exploring Design Choices for Building Language-Specific LLMs
- Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?
- WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
- An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model