When one model is trained on many languages at once, two kinds of knowledge compete for its capacity. The first is lexical: which word in Chinese or Finnish means what a given English word means. The second is structural: how words combine into phrases and clauses. The cross-lingual literature currently tells two stories that do not fit together. Controlled experiments find that reordering English into Japanese-like word order barely hurts transfer. Instead, transfer collapses when words map to many possible translations. Yet methods that inject explicit syntax into multilingual encoders report steady gains. This proposal tests a specific reconciliation: syntactic signals help multilingual models mainly by disambiguating the polysemous, many-to-many words that break cross-lingual alignment. If that is true, syntactic supervision should be light, and it should be concentrated on those ambiguous words rather than spread uniformly over every token.
Concretely, we would measure each word's translation entropy, meaning how spread out its aligned translations are in parallel text. We would then weight two auxiliary training signals by that entropy during all-languages-at-once training. One signal is a reduced set of dependency relations covering core noun phrases, prepositional attachments, and clause links. The other is a contrastive alignment of each word in context to its translation. The study asks three questions. Does this beat both uniform full-syntax supervision and purely lexical alignment on structure-sensitive tasks? Does it avoid the damage each of those does to the other axis? And do existing syntax-injection gains actually concentrate on high-entropy words?
Core Claim
The claim has two parts. We make the diagnostic part first because it is falsifiable on its own. First, the gains that current structure-injection methods achieve in cross-lingual settings are disproportionately located on tokens with high translation entropy. In other words, syntax acts partly as a regularizer on unstable lexical mappings, not only as a teacher of word order. Second, if that holds, a multilingual training objective that applies mild syntactic supervision where lexical mappings are ambiguous should do better than either ingredient alone. It should improve structured cross-lingual tasks without degrading lexical similarity, and it should improve lexical similarity without degrading parsing.
The second part is only worth building if the first part holds. The experiments are ordered so that a negative diagnostic result redirects the work. If gains are uniform across entropy buckets, the method collapses into "mild syntax everywhere", which is itself a new multilingual result, as explained below.
How This Sits Against Prior Work
The motivating tension comes from two groups of papers. Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations builds artificial languages from English by swapping script, word order, or vocabulary independently. It finds that only vocabulary swaps hurt: a 5.9% XNLI drop for a Chinese lexical swap. Swapping just the top third of high-entropy words does nearly as much damage as swapping everything. That study uses an English teacher distilled into a single student, and it evaluates on classification, where bag-of-words cues may suffice. It never tests joint multilingual training or structured prediction.
On the other side, two papers inject structure and report cross-lingual gains, but neither separates syntax learning from lexical regularization. Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer (LS-mBERT) biases mBERT attention with a graph network over dependency parses and adds dictionary code-switching. It gains up to 3.7 points on mTOP and 2.2 on WikiAnn. Structural Contrastive Pretraining for Cross-Lingual Comprehension (XLM-SCP) contrastively aligns subject-verb-object and entity backbones across parallel sentences. Our proposal differs from both in three ways. We ask where the structural gain lands. We replace random one-to-one dictionary swaps, which Ilani et al. implicitly show are the worst case for polysemous words, with alignment in context. And we train all languages jointly rather than transferring from English.
A second group of papers constrains how much syntax to use. How Much Syntactic Supervision is "Good Enough"? ablates grammar symbols in recurrent neural network grammars (language models that build parse trees as they generate). Keeping only noun phrases, prepositional phrases, and subordinate clauses scores 84.59% on SyntaxGym, beating full supervision on five of six test categories. Syntactic Structure Distillation Pretraining For Bidirectional Encoders finds that distilling full syntax into BERT helps parsing but gives mixed GLUE results. Both are monolingual English. Meanwhile the multilingual work, UDify and the SRL syntactic-supervision study, supervises full Universal Dependencies trees everywhere. We carry the "mild supervision" finding into the multilingual setting, which to our knowledge has not been tested.
A third group supplies the lexical method and shows the untested side effect. BabelBERT contrastively specializes mBERT and XLM-R on BabelNet synonyms across 50 languages, raising XLM-R's Multi-SimLex Spearman correlation from 5.9 to 34.2. LexFit does the same with small seed dictionaries. Neither measures parsing or compositional behavior afterwards. The monolingual literature (Sadeq et al., Shalaby et al.) suggests that word- and entity-level optimization erodes grammatical competence. We reuse their contrastive losses, but apply them to words in context and target them by entropy.
Finally, the shared-private BiLSTM parser and multilingual self-attention constituency parsing show that sharing structure while separating lexicons helps, and that full sharing costs high-resource languages. Our approach does not add modules. It tests whether a reallocation of training signal can achieve part of what architectural separation achieves.
Research Program
The base is XLM-R Base with continued joint training on a fixed language set: English, German, French, Spanish, Finnish, Chinese, Japanese, Hindi, and Thai. This set was chosen so that every language appears in at least one of the evaluations: UD, the Universal Proposition Bank, Multi-SimLex, mTOP, or MCWQ-R.
We compute translation entropy per word type and language pair from word-aligned bitext, using the released tooling from Ilani et al. (roy54x/Lexical_semantics_in_cross-lingual_transfer). We then average across the languages each word aligns into, giving a score $H_i$ for token $i$. Silver dependency parses provide the syntactic targets. The "mild" relation set is the UD analogue of noun-phrase/prepositional-phrase/subordinate-clause supervision: core arguments, obliques and case markers, nominal modifiers, and clausal complements and adjuncts. All other arcs are masked.
The training objective is
$$ \mathcal{L} = \mathcal{L}_{\text{MLM}} + \sum_i w(H_i)\,\big(\alpha\,\mathcal{L}^{\text{syn}}_i + \beta\,\mathcal{L}^{\text{align}}_i\big), \qquad w(H_i) = \frac{H_i}{\bar{H}}. $$
Here $\mathcal{L}_{\text{MLM}}$ is the usual masked-word loss. $\mathcal{L}^{\text{syn}}_i$ is a biaffine head-and-label loss for token $i$, restricted to the mild relation set. $\mathcal{L}^{\text{align}}_i$ is an InfoNCE contrastive loss that pulls token $i$'s contextual vector toward its aligned token in the parallel sentence and pushes it away from in-batch negatives. $\bar{H}$ is the corpus mean entropy, and $\alpha$ and $\beta$ are scalar weights tuned on development data. Setting $w\equiv 1$ recovers uniform supervision. That setting is the ablation that isolates the entropy-targeting idea.
Downstream, every model is fine-tuned by pooling task data across all available languages. XGLUE shows this multi-language setup beats English-only fine-tuning, so it is the right regime for a joint-training claim.
The analysis component comes first and costs little. We reimplement the LS-mBERT and XLM-SCP recipes on the same backbone and language set. We then bucket test-set errors on mTOP slots, Universal Proposition Bank arguments, and WikiAnn entities by the entropy of the words in the gold span. The question is whether their improvement over plain XLM-R grows with entropy.
The second analysis extends Ilani et al.'s controlled manipulations to joint training. We add a word-order-only artificial language and a lexical-only artificial language to the joint training mix. We then evaluate on structured tasks translated into those languages, in addition to XNLI. This tests whether word-order "immunity" survives outside classification.
Baselines
The naive reference is XLM-R Base continued on the same parallel and monolingual data with masked-language modeling only, then fine-tuned jointly on each task. Every other system must beat it, and it controls for the extra training tokens.
The strongest structural competitor is full UD multitask supervision in the style of the SRL syntactic-supervision study and UDify. It is strongest because it is the documented winner for cross-lingual semantic role labeling: that paper reports zero-shot Universal Proposition Bank average Argument-F1 rising from 51.91% to 58.52% for mBERT with full syntax. We run it with identical data, steps, and a matched hyperparameter search budget, differing only in using all UD relations and uniform weights. Beating it means higher Argument-F1 on the Universal Proposition Bank and higher mTOP exact match, together with a smaller drop on Multi-SimLex. For reference, XLM-R figures under our joint-fine-tuning protocol are not reported on Lacuna, so we report them fresh.
The strongest lexical competitor is a BabelBERT-style contrastive specialization (BabelBERT). We run it both with its original BabelNet constraints and with our contextual alignment loss at uniform weight and no syntax. On lexical similarity the bar is its reported XLM-R Multi-SimLex Spearman of 34.2. We do not expect to beat that. We expect to stay within a few points of it while not losing UD labeled attachment score, the parsing metric that it was never tested on.
The strongest combined competitor is our reimplementation of LS-mBERT and XLM-SCP on XLM-R Base. XLM-SCP reports MLQA F1 of 65.14 and WikiAnn F1 of 63.39 over XLM-R's 63.24 and 61.35. LS-mBERT reports only relative gains, up to 3.7 on mTOP and 2.2 on WikiAnn, and no absolute XNLI or PAWS-X numbers were found on Lacuna. The published numbers come from English-source transfer, so we compare against our reruns under the joint protocol.
The isolating ablations form a two-by-two grid: mild versus full relations, crossed with entropy-weighted versus uniform, plus syntax-only and alignment-only arms. We also include one control that weights by a random permutation of $H_i$. This separates an entropy-specific effect from the effect of simply reweighting tokens.
Why not just…?
Why not just scale up, for example XLM-R Large, and skip auxiliary signals? Scale has not closed either gap in the evidence we have. Sequence models score near 0% on COGS structural splits in the AM-parser study. Vanilla XLM-R sits at 5.9 Spearman on Multi-SimLex. Joint training across ten languages still hurt English and German parsing in Kitaev et al.. Still, the claim should be checked rather than assumed. We will run the naive baseline and the main method on XLM-R Large, and report whether the entropy-bucketed gap shrinks with size. A vanishing gap would be a real negative result.
Why not just do lexical specialization on the high-entropy words and drop the syntax? This is the most serious threat. If the diagnostic shows that structural gains live entirely on high-entropy tokens, entropy-targeted alignment alone might recover them. The alignment-only arm with entropy weights tests exactly this. If it matches the full method on mTOP, Universal Proposition Bank SRL, and PAWS-X, we will report that the structural channel is redundant under joint training. That would still resolve the Ilani-versus-LS-mBERT contradiction in favor of the lexical account. Our prior that syntax adds something comes from the SRL study's finding that syntax from a related language substitutes for target-language syntax, which a purely lexical signal cannot supply.
Why not just use full UD supervision everywhere, which already works for SRL? It works, but the monolingual evidence (Noji-style ablations, structure distillation) says full supervision taxes lexical reasoning. The SRL study itself notes that its benefit shrinks as target data grows. The full-versus-mild arms, together with a Multi-SimLex and G-BLI evaluation of every model, measure that tax directly in the multilingual case.
Why not just use dictionary code-switching, which is LS-mBERT's lexical channel? Random one-to-one substitution is precisely what fails for high-entropy words in Ilani et al. The code-switch-only arm inside the LS-mBERT reimplementation lets us check whether contextual alignment outperforms it specifically in the high-entropy bucket.
Proposed Experiments and Decisive Checks
The structured evaluations are mTOP exact match, Universal Proposition Bank and Finnish PropBank Argument-F1 (including the 100-sentence Finnish regime), UD v2.3 labeled attachment score across the nine languages, PAWS-X accuracy, and MCWQ-R exact match for Chinese and Japanese. No reported absolute MCWQ-R exact-match result was found on Lacuna, so we establish our own mT5-free encoder baseline there. The lexical evaluations are Multi-SimLex Spearman and G-BLI mean reciprocal rank. Each model therefore gets a point on a two-axis lexical-versus-structural plot, and the central figure is the trade-off frontier traced by all arms. We use three seeds per arm and paired bootstrap tests per language.
Three results decide the project. First, the entropy-bucketed error analysis of the LS-mBERT and XLM-SCP reruns should show gains increasing across entropy terciles. A flat profile falsifies the regularization hypothesis. In that case we drop entropy weighting and report mild-versus-full syntax as the multilingual contribution.
Second, the entropy-weighted mild model should beat the uniform full-syntax baseline on Universal Proposition Bank Argument-F1 and mTOP. It should do so while staying within two Spearman points of the alignment-only model on Multi-SimLex, and it should beat the random-permutation weighting control. Winning without beating that control means the effect is generic reweighting, not an entropy effect.
Third, in the joint-training version of Ilani et al.'s manipulations, the word-order-only language should show a clearly larger penalty on SRL and mTOP than on XNLI. That would demonstrate that the earlier "word order does not matter" finding is specific to classification. It would tell future multilingual work which kind of structural signal is worth paying for.
References
- Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations
- Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer
- Structural Contrastive Pretraining for Cross-Lingual Comprehension
- How Much Syntactic Supervision is “Good Enough”?
- Syntactic Structure Distillation Pretraining for Bidirectional Encoders
- 75 Languages, 1 Model: Parsing Universal Dependencies Universally
- On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role Labeling
- Massively Multilingual Lexical Specialization of Multilingual Transformers
- Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
- Unsupervised Improvement of Factual Knowledge in Language Models
- Beyond word embeddings: learning entity and concept representations from large scale knowledge bases
- A Simple yet Effective Joint Training Method for Cross-Lingual Universal Dependency Parsing
- Multilingual Constituency Parsing with Self-Attention and Pre-Training
- XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation
- Compositional generalization with a broad-coverage semantic parser