Paraphrase and inference models often take a shortcut: if two sentences share most of their words, the model calls them equivalent. Adversarial datasets break this habit in English by including pairs that share every word but differ in meaning, such as "flights from New York to London" versus "flights from London to New York." When an English-trained multilingual model is applied without any further training to Chinese, Japanese or Korean, though, its accuracy on the multilingual version of these pairs still lags behind European languages. This note argues that part of that gap is a units problem. In English, the overlap shortcut the model learns is overlap between subword tokens that roughly match words. In scripts that the tokenizer breaks into single characters, the same shortcut fires on overlap between character shards. Those shards are shared far more often and carry much less meaning.
The proposal is to make the model and its debiasing signal agree on a cross-lingually stable unit. A small character-aware layer composes fragments into word-sized units below the sentence encoder, a hierarchy of character, then word, then sentence. During English training, a deliberately shallow "bias model" sees only word-unit overlap statistics, and the main model is trained to explain what that bias model cannot. The bet is that debiasing transfers to a new language only when the bias being removed in English is measured in the same units as the bias the model will exploit in the target language. That bet is testable, and it can fail cleanly.
Core Claim
Lexical-overlap debiasing learned on English transfers zero-shot to over-fragmented scripts only if overlap is defined over word-like units rather than tokenizer pieces or bytes. The prediction is specific. On PAWS-X, word-unit debiasing should raise accuracy on non-obvious negatives (high-overlap pairs that are not paraphrases) in zh/ja/ko more than subword-unit debiasing does, with little difference in fr/es/de. In those European languages, subwords and words roughly coincide.
Two Senses of "Overlap," and Why the Literature Seems to Disagree
Two kinds of overlap are routinely conflated, and separating them is what makes this proposal coherent. The first is cross-lingual vocabulary overlap: whether languages share tokens. Here the evidence says sharing is not needed. Cross-Lingual Ability of Multilingual BERT: An Empirical Study builds a "Fake-English" with zero shared characters and loses only 1.4 XNLI points transferring to Spanish. Tokenization Impacts Multilingual Language Modeling finds that vocabulary sharing helps sentence-level tasks but hurts tagging. It also shows that fixed vocabulary budgets over-fragment some languages, a finding that motivates this proposal.
The second is within-pair overlap: how much a premise and hypothesis share. This is the shortcut. PAWS shows BERT falling from over 90% on Quora pairs to 33–40% on scrambled pairs, and recovering to 85% with 12,000 adversarial training examples. It Is Not Easy To Detect Paraphrases reports a popular sentence encoder at 1.9% on its negation-and-antonym test, against 33.3% for random guessing. Aiming beyond the Obvious gives the measurement tool we reuse. It splits pairs by Jensen-Shannon divergence over token distributions into obvious and non-obvious positives and negatives, and reports a score, $F1_n$, on the non-obvious ones. That paper computes divergence over a single tokenization. We treat the unit as the experimental variable.
The two senses meet in fragmented scripts. Over-fragmentation, documented by Limisiewicz et al., inflates within-pair overlap when it is measured in tokens. Nobody has asked whether this changes which pairs a transferred model finds "obvious." PAWS-X motivates the question directly. Its zero-shot setting already trains on adversarial English data, yet CJK accuracy sits around 83–84% while other languages do better. So English adversarial data alone does not carry the fix across scripts.
Two other papers constrain the design. Assessing the Role of Lexical Semantics in Cross-lingual Transfer finds that script substitution by itself barely hurts transfer, and that lexical-semantic mismatch is the main cost. Our hypothesis is therefore not "scripts are hard." It is narrower: script-dependent fragmentation miscalibrates the overlap shortcut. Separately, SHINE shows that multi-level interaction helps zero-shot information extraction. It depends on constituency parsers, however, which we avoid. Our hierarchy needs only word boundaries.
Research Program
Phase one: diagnosis. This phase is cheap and decides whether the method is worth building. We fine-tune mBERT and XLM-R on English PAWS-Wiki and evaluate zero-shot on the human-translated PAWS-X test sets. Each test pair is partitioned three times with the divergence criterion of Peinelt et al.: once over tokenizer subwords, once over UTF-8 bytes, and once over word units. Word units come from whitespace for most languages and from off-the-shelf segmenters for Chinese and Japanese; using unlabeled segmentation tools does not break the zero-shot condition, because no task labels are involved.
Two hypotheses are tested. H1 is that, in CJK, the subword partition disagrees with the word partition much more than in European languages. H2 is that model errors are better predicted by subword overlap than by word overlap in CJK, measured by logistic regression of error on each overlap statistic, with a likelihood comparison. If H2 fails, the units story is wrong and Phase two is not run.
Phase two: the method. Between the multilingual encoder's subword outputs and the pair classifier, we insert a composition tier. A one-dimensional character convolution with max-pooling and highway gates reads the characters of each word unit, in the style of Character-based Neural Machine Translation and Towards Sub-Word Level Compositions. Its output is concatenated with attention-pooled subword states for that unit, using a word-level context vector as in Hierarchical Attention Networks. A light transformer layer over word units then feeds a sentence-pair head. Here the hierarchy is character, then word, then sentence-pair, not a document hierarchy. Document-level reading comprehension, as in SAM, lacks a multilingual test set in our file, so we leave it out rather than stretch the claim.
Training on English uses a product of experts. A bias-only classifier $p_b$ sees only overlap features $o_u(x)$ of pair $x$ at unit $u$: divergence, shared-unit ratio, and order-preserving shared n-grams. The main model's distribution $p_m$ is trained through the combined prediction
$$p(y \mid x) \propto p_m(y \mid x)\, p_b(y \mid o_u(x)),$$
where $y$ is the paraphrase label. At test time only $p_m$ is used. The combination absorbs the overlap-predictable part of the label, which pushes $p_m$ to rely on order and composition instead. The central manipulation is $u \in \{\text{byte}, \text{subword}, \text{word}\}$, crossed with the presence or absence of the composition tier.
Baselines
The naive reference point is mBERT or XLM-R fine-tuned on English PAWS-Wiki and applied zero-shot, which is the PAWS-X zero-shot protocol. For mBERT, the file lists the relevant reported figures only for the Merged setting: 87.2% average accuracy across non-English languages in PAWS-X, with CJK around 83–84%. No reported result was found on Lacuna for XLM-R zero-shot on PAWS-X, so we will run it ourselves. Every system uses the same encoder checkpoint, the same English training data, the same number of hyperparameter trials selected on English dev only, and five seeds. Accuracy is reported overall and on the non-obvious subsets.
The strongest existing method is translate-train, the PAWS-X "Merged" regime: English plus machine-translated target training data, with 87.2% non-English accuracy for mBERT. This violates strict zero-shot, but it is the number a practitioner would compare against. We rerun it with the same encoder and budget. Beating it means matching its overall accuracy within one point while exceeding it on CJK non-obvious negatives. A real win for the zero-shot method would be exceeding it outright on CJK $F1_n$.
The subword-unit debiasing baseline is identical to the method except that $u$ is the tokenizer's subword unit and there is no composition tier. This is the naive way anyone would port English debiasing, so it is the most important competitor for the core claim.
Two ablations isolate the ingredients. The composition tier without debiasing tests whether the hierarchy helps alone. Word-unit debiasing without the tier tests whether units in the bias model alone suffice. A byte-unit bias model tests the extreme of fragmentation.
Why not just…?
Why not just translate the training data? Translate-train is a serious threat on PAWS-X, because the target languages are high-resource and the translations are of good quality. The kinship-gap study, Zero-Shot vs. Translation-Based Cross-Lingual Transfer, shows machine translation collapsing fine distinctions; GPT-3.5 drops from 98.70% on oracle Farsi to 66.60% with NLLB translations. That is a different task, though, so it does not settle PAWS-X. We run translate-train under the same budget and report it honestly. If it dominates on CJK non-obvious negatives, our contribution shrinks to the diagnosis plus a method for languages without good translation systems. We also stack the two to check whether the gains are complementary.
Why not just add more adversarial English data? The PAWS-X zero-shot setting already does this, and CJK still lags. Our naive baseline therefore is this alternative. The open question is whether more data of the same kind fixes a unit mismatch. We test it by scaling the English adversarial set and plotting CJK non-obvious-negative accuracy against data size, for both subword-unit and word-unit systems.
Why not just use bytes so every script gets equal treatment? Measuring Cross-lingual Transfer in Bytes shows that byte-level modeling equalizes budgets and transfers well. It does not test overlap shortcuts. Byte overlap is the finest and least meaningful unit of all, so our hypothesis predicts that byte-unit debiasing transfers worst. The byte-unit ablation tests this directly.
Why not just whiten the embeddings with BERT-flow? On the Sentence Embeddings from Pre-trained Language Models lowers the link between similarity and edit distance, from −50.49 to −28.01. The authors note, however, that a flow cannot add information the encoder lacks, and word order on PAWS is exactly such information. It also targets cosine-similarity embeddings rather than a fine-tuned cross-encoder. We treat it as out of scope rather than as a competitor.
Decisive Checks
The main test set is PAWS-X human-translated test data in all seven languages. The secondary test set is XNLI (XNLI), partitioned the same way to check whether gains extend beyond paraphrase. On XNLI, the gap between obvious and non-obvious accuracy also gives a direct reading on the Limisiewicz et al. "overlap helps semantics" finding: if the benefit of overlap lives mostly in obvious pairs, it is partly an artifact. English SemAntoNeg serves as a check that debiasing does not damage negation handling.
The idea is confirmed only if three things hold. First, H2 holds in Phase one. Second, word-unit debiasing beats subword-unit debiasing on CJK non-obvious negatives by more than seed variance across five seeds, while staying within one point on European languages. Third, the byte-unit system is worst. If word and subword units tie, the units hypothesis is wrong, whatever the absolute accuracy. To probe the segmentation risk flagged by HAN-style models, we also perturb CJK word boundaries at test time with a different segmenter and with random merges. A method whose gains vanish under a second segmenter is not useful for low-resource scripts, and we will report that as a failure.
References
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
- PAWS: Paraphrase Adversaries from Word Scrambling
- It Is Not Easy To Detect Paraphrases: Analysing Semantic Similarity With Antonyms and Negation Using the New SemAntoNeg Benchmark
- Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets
- PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
- Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations
- Syntax-Augmented Hierarchical Interactive Encoder for Zero-Shot Cross-Lingual Information Extraction
- Character-based Neural Machine Translation
- Towards Sub-Word Level Compositions for Sentiment Analysis of Hindi-English Code Mixed Text
- Hierarchical Attention Networks for Document Classification
- Semantics Altering Modifications for Evaluating Comprehension in Machine Reading
- Zero-Shot vs. Translation-Based Cross-Lingual Transfer: The Case of Lexical Gaps
- Measuring Cross-lingual Transfer in Bytes
- On the Sentence Embeddings from Pre-trained Language Models
- XNLI: Evaluating Cross-lingual Sentence Representations