When Word Parts Speak Out of Turn: Sub-Lexical Meaning Leakage in Zero-Shot Cross-Lingual Transfer

Human readers of Chinese treat the meaningful parts of a character as silent until the whole character is recognized as a real word. A "foot" component primes foot-related meanings inside genuine characters, including characters whose overall meaning has nothing to do with feet. The same component primes nothing inside a well-formed but non-existent character. Multilingual language models work differently. They build a word's meaning bottom-up from subword pieces and have no step that checks whether the pieces form a real word before those pieces add their own meaning. We propose that this difference explains part of what goes wrong in zero-shot cross-lingual transfer, the setting where a model is fine-tuned on an English task and then applied directly to another language with no labeled examples. The specific hypothesis is that when a target-language word is opaque (its pieces mean something other than the whole, like Chinese 东西 "east + west" meaning "thing"), the pieces' standalone meanings leak into the representation that the English-trained task layer reads. The model then answers as if the word meant its parts.

The core claim has two parts. Transfer errors should cluster on opaque multi-piece words in proportion to measurable leakage, over and above the whole-word lexical ambiguity already known to hurt transfer. And a controlled manipulation that varies only piece-level opacity should reproduce that loss. The main contribution is a diagnostic and causal study: a model-side version of the human genuine-word, opaque-word, and pseudo-word paradigm, plus a synthetic-language experiment. A small intervention tests whether correcting representations only for validated, opaque words beats correcting all words. The intervention is secondary to the diagnostic and causal study.

What the two literatures give us, and what is missing

The psycholinguistic side motivates the hypothesis. The Phonograms' Genuine-Character Status showed behaviorally that foot radicals prime semantically inside genuine phonograms but not inside pseudo-characters. The deciding factor was whether the host was a real character, not whether the radical could stand alone as a character. Semantic radicals' semantic attachment to their composed phonograms used masked priming and found facilitation of roughly 50–80 ms even in opaque characters, with none in pseudo-characters. They call this "semantic attachment": a part's meaning activates only after its host passes a lexical check. Semantic Transparency of Radicals in Chinese Characters showed that native speakers share structured, dialect-invariant judgments of how transparent a part-to-whole relation is. That gives us a reason to expect opacity to be a stable property of words rather than rater noise. These studies cover only Chinese radicals, and mostly the foot category. Whether "gating by lexical status" applies to other writing systems, or to machines, is open.

The transfer side constrains the design. Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations distilled English RoBERTa into artificial languages and reported two results:

  • Swapping script or word order barely hurts transfer.
  • Raising translation entropy hurts it a lot. Translation entropy is the spread of a word's possible translations, measured over a word-alignment graph. Adopting Chinese lexical structure cost 5.9% XNLI accuracy and 0.092 in alignment cosine.

That paper treats words as atomic units, so piece-level effects cannot appear in it. This also helps resolve an apparent contradiction. Its script swaps were one-to-one glyph substitutions that left segmentation untouched. The real script gaps on MLQA (XLM at 54.8 F1 on Arabic and 48.8 on Hindi versus 68.0 on Spanish) come bundled with different segmentation and composition. Our manipulation separates exactly that bundled factor.

On the tokenization side, Evaluating Subword Tokenization found that "alien" (non-morphological) splits hurt English out-of-vocabulary generalization. Lexically Grounded Subword Segmentation chose splits whose pieces compose to the whole-word embedding. Both treat the problem as bad boundaries. Our hypothesis is different, because opacity leakage can occur even when the boundaries are perfectly morphological. 东西 splits into two genuine morphemes, and the problem is that those morphemes mislead. Zero-shot Reading Comprehension by Cross-lingual Transfer Learning documented the English-to-Chinese gap (78.82 vs 88.68 F1 on DRCD) but did not examine the encoder's internals.

One transposition is necessary and must be stated plainly. Common Chinese characters are single tokens in mBERT and XLM-R, so radicals are invisible to these models. Pseudo-characters cannot even be encoded in Unicode. The analogue of radical-in-character is therefore character-in-word for Chinese, and subword-in-word for Korean, Hindi, and Arabic. We test the principle of lexical gating, not the radical results literally.

Research Program

Leakage measure. For a target-language word $w$ split into pieces $p_1,\dots,p_k$, we define leakage at layer $\ell$ as

$$L_\ell(w) = \cos\big(h_\ell(w), \bar g(p_{1..k})\big) - \cos\big(h_\ell(w), g(w)\big),$$

where the terms are:

  • $h_\ell(w)$ is the model's contextual representation of $w$ at layer $\ell$, averaged over its pieces and over corpus occurrences.
  • $g(w)$ is the model's representation, in English contexts, of $w$'s dictionary gloss.
  • $\bar g(p_{1..k})$ is the mean of the corresponding English representations of the pieces' standalone glosses.

Positive leakage means the word "looks like" its parts more than its meaning in the space the English task head was trained on. We also compute opacity from dictionaries alone, as the dissimilarity between the word's gloss and its pieces' glosses. Opacity describes the language; leakage describes the model.

Model-side lexical gating test. We build three kinds of stimulus sets:

  • Genuine transparent words, whose parts match the whole.
  • Genuine opaque words, whose parts do not match the whole.
  • Pseudo-words: legal recombinations of real pieces that are not words. For Chinese these are unattested two-character strings, checked against a corpus word list.

Each item is placed in neutral sentence frames. The priming analogue is how much the item's representation moves toward English probes related to one of its pieces. Humans show piece activation in both kinds of genuine word and none in pseudo-words. A bottom-up encoder would activate pieces everywhere, and most strongly in pseudo-words, where no whole-word meaning competes. Where the model's pattern departs from the human one is itself the finding. We also run this test layer by layer, to see whether upper layers suppress piece meaning for genuine words the way lexical access does.

Error attribution. We take zero-shot errors on Chinese reading comprehension and inference and ask whether words in the question and near the gold answer carry higher leakage. The regression controls for word frequency, piece count, and whole-word translation entropy, the last computed with the released Lexical_semantics_in_cross-lingual_transfer code. Leakage is only interesting if it predicts errors after translation entropy is controlled.

Causal synthetic languages. We extend the distillation setup of Ilani et al. Every English word is rendered as a two-piece composite drawn from an artificial piece inventory, and the pieces also occur as standalone words with their own meanings. There are three conditions:

  • Transparent: the pieces' standalone meanings relate to the word's meaning.
  • Opaque: the pieces map to unrelated standalone words.
  • Atomic control: each word is a single novel token.

The word-level mapping is one-to-one in all three, so whole-word translation entropy is held at zero. Any gap between the opaque and transparent conditions on alignment cosine and XNLI transfer is attributable to piece-level opacity alone.

Validation-gated alignment (the intervention). We continue training the multilingual encoder briefly on unlabeled target text, adding a contrastive loss. The loss pulls the contextual representation of a target word toward its gloss's English representation, but only for spans that pass two checks: the span is in a target lexicon (the "validation" step), and its opacity is above a threshold. The model is then fine-tuned on English and evaluated zero-shot. The design question is whether restricting correction to validated opaque words matters, or whether any word-level alignment does as well.

Baselines

The naive reference point is standard zero-shot transfer: fine-tune mBERT, XLM-R, and mT0 on English SQuAD or MultiNLI and evaluate directly on the target language. The strongest reported number for our main Chinese setting is multi-BERT's 78.82 F1 on DRCD after SQuAD training, against 88.68 F1 with native Chinese training (Hsu et al.). We reproduce this number before claiming anything. For MLQA Chinese and for XNLI Chinese in absolute terms, no reported result was found on Lacuna for those specific cells, so we report our own reproduced baselines there.

The strongest competing method is word-level dictionary alignment in the style of LexFit, from Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders. It is contrastive tuning on 1k–5k translation pairs and improved bilingual lexicon induction by about 10 Precision@1. It targets the same lexical mismatch as our intervention with the same kind of resource. We run it with the identical dictionary, identical unlabeled target text, identical steps and learning-rate search, and three seeds. It differs from our intervention only in applying to all dictionary words.

Our key ablations use the same budget and change only which spans are aligned:

  • No validation step: all multi-piece spans, including pseudo-word spans.
  • Transparent words only.
  • Random words, matched in count.

Our intervention beats the strongest competitor only if three things hold: it exceeds all-word alignment on DRCD, MLQA-zh, and XNLI-zh, with a paired bootstrap interval above zero across seeds; the gain is concentrated in the opaque-word stratum of test items; and the "no validation" ablation does worse. If all-word alignment matches it, the gating idea adds nothing as an intervention, and we report that outcome.

Why not just…?

Why not just translate the training data (or the test inputs)? Hsu et al. found that translate-train lowered DRCD performance relative to pure zero-shot transfer. XNLI reported translate-test reaching above 70% but depending on high-quality machine translation. We run translate-train on DRCD as a reference. Translation also answers a different question: it replaces the encoder's handling of opaque words rather than diagnosing it. The leakage analysis stands either way.

Why not just use morphology-aware tokenization or subword sampling? These fix where words are split, and the evidence behind them concerns alien boundaries (Batsuren et al.; Libovický and Helcl). Opaque Chinese compounds are already split at genuine morpheme boundaries, and the leakage arises because those morphemes carry the wrong meaning. The synthetic experiment settles this directly, because every opaque composite there has clean boundaries. If the opaque condition still loses transfer, better segmentation cannot be the whole fix. We also include subword-regularized fine-tuning as a baseline on the non-Chinese MLQA languages, where boundaries do vary.

Why not just scale up or instruction-tune? Crosslingual Generalization through Multitask Finetuning found that mT0-13B often beat BLOOMZ-176B, so parameter count alone does not settle transfer quality. Scale is still a real threat to the hypothesis: larger models might suppress leakage in their upper layers. We therefore run the gating test and leakage measurement on mT0 at several sizes. If leakage falls to zero with size and error attribution disappears, the hypothesis is limited to small encoders, and we will say so.

Why not just align words with a bilingual dictionary? This is the most serious alternative, and it is why all-word LexFit-style alignment is our main baseline under matched budgets. Ilani et al. report that high-entropy words keep hurting transfer even with aligned lexicons, which leaves room for a targeted method, but the comparison is empirical and we do not presume the outcome.

Decisive Checks

Three outcomes would decide the question. First, in the synthetic languages with whole-word translation entropy held at zero, the opaque condition should transfer measurably worse than the transparent condition on XNLI and alignment cosine; if it does not, piece-level opacity has no causal role and the hypothesis fails. Second, on natural Chinese data, leakage should predict zero-shot errors on DRCD and MLQA-zh after controlling for translation entropy and frequency, and the effect should replicate on Korean and Hindi subword-level words. Third, the model-side gating test should show whether encoders activate piece meanings in pseudo-words as strongly as in genuine words, unlike humans. This result is informative whichever way it goes: non-gating supports the leakage account, and human-like gating in upper layers would point to the task head as the source of the failure. The intervention's comparison against all-word alignment then shows whether acting on this diagnosis is worth more than generic lexical alignment.