When building multilingual natural language processing models, it is tempting to assume that cross-lingual transfer is primarily a matter of vocabulary translation. If a model can map the English word "dog" and the French word "chien" into the same vector space, we expect it to seamlessly transfer downstream tasks from one language to another.
However, as Mikel Artetxe et al. demonstrate in On the Cross-lingual Transferability of Monolingual Representations, cross-lingual transfer performance degrades significantly when the target language exhibits high syntactic divergence from the source language. The core bottleneck is not vocabulary, but structure. Even when multilingual embeddings successfully align word meanings, they often fail to account for the divergent ways those words are organized into sentences (Tao Ji et al. in Word Reordering for Zero-shot Cross-lingual Structured Prediction). This phenomenon, known as cross-lingual structural divergence, creates a fundamental gap between semantic generalization and syntactic generalization.
How Models Overfit to Source Syntax
Why do state-of-the-art neural networks struggle with structural divergence? The answer lies in how these models process sequences. During training on a high-resource source language (typically English), neural models do not just learn word meanings; they internalize the specific word order and grammatical conventions of that language, such as Subject-Verb-Object (SVO) ordering or specific head-modifier directions.
As M. Saiful Bari et al. point out in Zero-Resource Cross-Lingual Named Entity Recognition, neural models become overly specialized to the word order and syntactic conventions of their training language. For instance, sequence labeling and information retrieval models often rely heavily on local n-gram patterns and positional encodings (Sean MacAvaney et al. in Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning).
When a model trained on an SVO language is evaluated zero-shot on a Subject-Object-Verb (SOV) language, its internal positional expectations are violated. The encoder expects a specific syntactic structure that is missing, resulting in noisy contextual representations and decoding failures (V. Rudra Murthy et al. in Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages). This structural overfitting means that source-language data containing word orders non-existent in the target language can act as pure noise, degrading cross-lingual adaptation (Anders Søgaard in Data point selection for cross-language adaptation of dependency parsers).
Strategies for Bridging the Structural Gap
To overcome this typological gap, researchers have developed several distinct strategies designed to normalize or decouple syntax across languages.
Explicit Syntactic Reordering
One intuitive solution is to physically rearrange the words of one language to match the grammatical structure of another. By transforming the target language's word order to mimic the source language (or vice versa), the model's internalized structural expectations are preserved.
For example, Lu Liu et al. introduce a method in Cross-Lingual Dependency Parsing by POS-Guided Word Reordering that uses a part-of-speech guided language model to reorder source sentences, creating synthetic training data that matches the target language's syntax. This directly mitigates the typological gap that prevents a parser trained on one word order from generalizing to a language with a fundamentally different arrangement.
Delexicalization and Universal Syntax
Another prominent approach involves stripping away language-specific lexical features entirely, forcing the model to focus purely on abstract grammar. In Multi-Source Transfer of Delexicalized Dependency Parsers, Ryan McDonald et al. demonstrate that removing specific words and training solely on universal part-of-speech tags allows syntactic knowledge to transfer across languages.
However, even delexicalized transfer faces constraints when typological distances are vast (Lauriane Aufrant et al. in Zero-resource Dependency Parsing: Boosting Delexicalized Cross-lingual Transfer with Linguistic Knowledge). To address this, frameworks like Selective Sharing for Multilingual Dependency Parsing by Tahira Naseem et al. explicitly decouple the parsing process into universal "selection" (identifying which words depend on each other) and language-specific "ordering" (determining where those words sit sequentially).
Target-Aware and Counterfactual Constraints
Rather than changing the input text, some approaches modify how the model reasons during training or inference. In Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing, Nanyun Peng et al. inject target-language typological knowledge directly into the inference phase via corpus-wise constraints, preventing the model from applying incorrect source-specific grammatical patterns.
Alternatively, the authors of COSY: COunterfactual SYntax for Cross-Lingual Understanding utilize a counterfactual training objective to force multilingual models to rely on language-agnostic structural features rather than treating syntax as a redundant or ignorable signal.
Ultimately, this body of research underscores that true cross-lingual understanding requires moving beyond word-level semantic alignment. By explicitly accounting for syntactic divergence—whether through token reordering, structural delexicalization, or target-aware inference constraints—models can successfully generalize across deeply disparate language families.