Semantic Transport Costs for Low-Overlap Cross-Tokenizer Distillation

When a large "teacher" language model is distilled into a smaller "student" that splits text into different pieces (a different tokenizer), the teacher's next-token probabilities must be moved onto a vocabulary they were never defined over. Current methods move this probability mass in one of two ways. Some match only tokens whose surface strings coincide. Others treat every mismatch as equally wrong, so putting the teacher's confidence in "kitten" onto the student's "cat" costs exactly as much as putting it onto "carburetor." We propose to make that cost reflect meaning. The penalty for moving mass from a teacher token to a student token becomes their distance in a shared embedding space. That space is built by learning a projection from the teacher's embedding geometry into the student's, much as vision-language models learn a projector from image features into a language model's word space.

This matters most where existing methods fail. When two vocabularies share fewer than about a quarter of their tokens, standard cross-tokenizer distillation losses produce negative transfer: the student ends up worse than one trained by plain fine-tuning on the teacher's text. In that regime exact string anchors are scarce, and the teacher's knowledge is spread over tokens the student cannot match literally. Our bet is that a meaning-aware cost recovers most of what string matching discards. We also bet the gain can be measured as a function of vocabulary overlap.

Core Claim

A cross-tokenizer distillation loss whose transport cost is the cosine distance between projected token embeddings will beat anchor-based and uniform-cost methods. The margin should widen as the share of exactly shared tokens falls. The improvement must come from the semantic content of the cost, not from the extra smoothing that any non-uniform cost introduces. A cost matrix with the same entries randomly permuted should therefore recover none of it.

The prior work divides cleanly into three groups. Two papers motivate the gap by fixing the cost to be meaning-blind. ULD frames cross-tokenizer logit distillation as optimal transport, meaning the cheapest way to move one probability distribution onto another under a per-pair cost. It sets that cost to 1 for every mismatched pair to obtain a fast closed form, and it explicitly leaves non-uniform semantic costs unexplored because of compute. E-SCOT aligns positions by character spans and compares sorted probability profiles, so it ignores which words carry the probability.

ACTD supplies our scaffolding and our strongest competitor. It aligns sequences on character offsets and distills only the teacher's top-$k$ "anchor" tokens. It adds an L1 penalty on student mass outside the anchors, and it shows that transferring the full long tail injects noise. Its handling of unshared tokens is the specific weakness we target. It decodes the teacher token to text and assigns all of that token's probability to the student's first subtoken, discarding the rest of the word.

Two papers supply method ingredients and constraints. Worse WER, but Better BLEU? aligns speech-model hidden states to word vectors. It shows that a temperature-scaled cosine softmax helps where plain cosine distance fails, which it attributes to "hubness": a few vectors become nearest neighbours of disproportionately many others. That warns us that a naive cosine cost will be dominated by hub tokens. The same paper also shows that meaning-preserving substitutions can raise downstream quality while lowering exact surface agreement, which is the trade-off we expect to see. Deep Pre-Alignment shows that a shallow projector leaves foreign features geometrically misplaced in a language model's space. That is the reason we fit and correct the teacher-to-student projection explicitly rather than assuming the two embedding spaces are comparable.

Research Program

The loss operates per aligned position. ACTD-style monotone character-offset alignment produces matched positions. At each one we take the teacher's top-$k$ tokens with renormalized probabilities $p$ and the student's top-$m$ candidates plus the first subtokens of the teacher candidates' strings, with probabilities $q$. The cost between teacher token $i$ and student token $j$ is

$$C_{ij} = \begin{cases} 0 & \text{if the strings of } i \text{ and } j \text{ are identical,} \\ 1 - \mathrm{csls}(W e^T_i,\, e^S_j) & \text{otherwise,} \end{cases}$$

where $e^T_i$ and $e^S_j$ are the teacher's and student's output-embedding rows, and $W$ is an orthogonal map. $W$ is fit by Procrustes alignment, a closed-form rotation that best superimposes paired points, using the shared-string tokens as pairs. The function $\mathrm{csls}$ is cosine similarity minus each vector's mean similarity to its nearest neighbours, a standard correction that stops hub tokens from looking close to everything. The training objective is

$$\mathcal{L} = \mathrm{OT}_\varepsilon(p, q; C) + \lambda \,\lVert q_{\bar A} \rVert_1 + \mathcal{L}_{\mathrm{CE}},$$

where $\mathrm{OT}_\varepsilon$ is entropy-regularized transport solved with a few Sinkhorn iterations, $q_{\bar A}$ is student mass outside the candidate set (ACTD's residual penalty), $\lambda$ is its weight, and $\mathcal{L}_{\mathrm{CE}}$ is ordinary cross-entropy on the reference text.

Because the transport runs over a $k \times m$ block with $k, m \le 32$, the cost ULD cites as prohibitive becomes a small batched matrix computation. We will report wall-clock overhead per training step against ACTD.

Three design questions will be studied directly. The first is where the geometry should come from. We compare the Procrustes-projected teacher space, the student's own input embeddings applied to the re-tokenized teacher string, and an external frozen word encoder. The second is whether hubness correction is needed: CSLS against raw cosine, mirroring the cosine-softmax finding. The third is how the method behaves as anchors vanish, since $W$ itself is fit on shared tokens.

Baselines

The naive reference point is supervised fine-tuning of the student on teacher-generated text, with no logits used. CTPD reports it at 64.54 average against 67.42 for logit-aware transfer in a different setting. At low overlap, the file reports that distillation can fall below this baseline, so it is the bar that matters there.

The strongest existing method is ACTD. It reports an average math score of 27.8 for DeepSeek-R1-Distill-Llama-8B into Qwen3-1.7B, against 22.9 for ULD. We will reproduce ACTD in exactly that teacher-student pair, on its OpenThoughts-114k training data and its math suite. Beating it means a statistically reliable gain over our reproduced ACTD number, using the same top-$k$, the same number of training tokens and the same hyperparameter search budget. We will report the reproduction alongside the published 27.8.

ULD and E-SCOT are run in the same pipeline as additional distribution-matching competitors. For E-SCOT the file gives only relative gains (+4.67% for Qwen2-0.5B and +4.32% for Gemma-2-2B over the next best approaches), not absolute scores.

The ablation that isolates the key ingredient keeps our Sinkhorn solver, candidate sets and residual penalty fixed and swaps only the cost matrix. The variants are uniform cost (ULD-style), the same semantic cost with entries randomly permuted within each block, character edit distance between token strings, and the full semantic cost. If the permuted cost matches the semantic cost, the gain is generic smoothing and the claim fails.

Why not just…?

Why not just fine-tune on teacher text and skip logits? In high-overlap pairs, logit methods already beat it. Examples are CTPD's 67.42 against 64.54, and ULD's reported 2.30-point gain on extractive QA with half the data. The open question is the low-overlap regime, where the file says distillation falls below this baseline. Our overlap sweep includes text-only fine-tuning at every point. If semantic transport does not climb back above it at the lowest overlap levels, the method does not solve the problem it targets.

Why not just embed the teacher's token string with the student's own subtoken embeddings and skip the projection? This is a real threat because it is cheaper and needs no shared anchors. It is run as one of the three geometry variants. We expect it to be weakest for multi-subtoken words, whose averaged subtoken vectors encode spelling more than meaning. The comparison will be broken down by the number of student subtokens per teacher token, so the evidence can say where, if anywhere, the projection is worth its cost. If string re-embedding ties, we adopt it and say so.

Why not just label-smooth or raise the distillation temperature? Any non-uniform cost spreads student mass, and smoothing does that too. Label smoothing and teacher temperature are tuned for ACTD under the same budget. The permuted-cost ablation separates semantics from spreading.

Why not just decode byte by byte with ByteSampler or CharED? These are exact inference-time ensembling methods, not distillation methods. They need multiple forward passes per subword token, and ByteSampler depends on BPE merge structure. They do not produce a standalone student, so they answer a different question. We do not run them as baselines.

Proposed Experiments

The primary experiment reproduces ACTD's setting: DeepSeek-R1-Distill-Llama-8B into Qwen3-1.7B on OpenThoughts-114k, evaluated on AIME24, AIME25, AMC23, MATH-500 and Minerva Math, averaged as in ACTD. A secondary instruction-following setting follows E-SCOT, with Dolly-15k for training and MMLU, ARC, HellaSwag and TruthfulQA for evaluation, using Qwen2-0.5B and Gemma-2-2B students. A third uses GSM8K, where Cross-Tokenizer Likelihood Scoring reports 55.6% for its partial-KL method, as an external point of reference.

The decisive check is a controlled overlap sweep. Natural tokenizer pairs confound overlap with everything else, so we artificially withhold exact-string anchor status from a random fraction of shared tokens. Those tokens are treated as unshared by every method, and the Procrustes fit uses only the retained anchors. We sweep effective overlap from the natural level down to 5%. The prediction is that ACTD's and ULD's advantage over text-only fine-tuning collapses below roughly 25% overlap, while semantic transport degrades more gently. The permuted-cost ablation should track ACTD rather than the full method. A flat gap across the sweep, or a permuted-cost curve matching ours, falsifies the claim.

A second check tests the "worse surface, better meaning" prediction. On held-out text we measure top-1 token agreement between student and teacher, and the rate at which student errors are near-synonyms of the teacher's choice, judged by the external encoder. Semantic transport should show lower exact agreement but a higher synonym share and higher task accuracy. Finally, we report per-step training overhead and the sensitivity of results to $k$, $m$, the Sinkhorn regularization $\varepsilon$ and the CSLS neighbourhood size. Projecting teacher signal through a learned map into a student whose input space differs is the same problem a vision-language connector solves, so extending the method to vision-language teachers and students with different tokenizers is deferred until the text result holds.