Abstract: Cross-lingual alignment, the meaningful similarity of representations across languages in multilingual language models, has been an active field of research in recent years. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field. We present different understandings of cross-lingual alignment and their limitations. We provide a qualitative summary of results from a large number of surveyed papers. Finally, we discuss how these insights may be applied not only to encoder models, where this topic has been heavily studied, but also to encoder-decoder or even decoder-only models, and argue that an effective trade-off between language-neutral and language-specific information is key.
Introduction
In the field of Natural Language Processing (NLP), the goal of "zero-shot cross-lingual transfer" is paramount. This capability allows a model trained on a task in a high-resource language, such as English, to perform effectively on a lower-resource language, like Hindi, without additional training. This transfer is made possible by cross-lingual alignment: the phenomenon where a model's internal representations of semantically equivalent terms (e.g., "apple" and "pomme") are mathematically similar.
Despite its importance, "alignment" is often used loosely in literature. This survey addresses the lack of a unified framework by defining alignment, exploring measurement techniques, and identifying the most effective methods for improving it across various model architectures.
Taxonomy of Alignment Strategies
Rather than proposing a single algorithm, this survey categorizes five years of research (2019–2023) into a taxonomy based on training objectives and data requirements. Methods are broadly divided into those that adapt existing models and those trained from scratch.
Key strategies include:
- Parallel Data Objectives: Utilizing translated sentence or word pairs to force representations into proximity.
- Contrastive Learning: A powerful technique that pulls "positive" translation pairs together while pushing "negative" dissimilar pairs apart. This is particularly effective at reducing "hubness"—a geometric phenomenon where certain vectors become neighbors to many others, degrading retrieval accuracy.
- Transformation of Representations: Post-hoc adjustments, such as rotating the embedding space or "mean-centering" language-specific subspaces to increase overlap.
- Adapter Tuning: Integrating small, trainable modules into frozen pre-trained models to specialize them for specific languages or tasks.
The survey also distinguishes between two conceptual "Views" of alignment:
- View I (Similarity-based): Focuses on whether similar meanings are close together across the entire vector space.
- View II (Subspace-based): Investigates whether a simple linear "head" can identify language-neutral, task-relevant directions within the data.
Benchmarks and Evaluation
The effectiveness of these strategies is typically measured across several standard multilingual benchmarks:
- XNLI: Cross-lingual Natural Language Inference (classification).
- UD-POS: Universal Dependencies Part-of-Speech tagging (sequence labeling).
- PAWS-X: Paraphrase Adversaries from Word Scrambling (sentence similarity).
- MLQA: Multilingual Question Answering.
- Tatoeba & BUCC: Sentence retrieval and bitext mining, measuring the model's ability to identify translations.
Key Insights and Results
Research indicates that "strong" alignment is not always the ultimate goal. While perfect overlap of translations sounds ideal, it can lead to "over-alignment," where the model loses the unique nuances of individual languages. In generative models, this can even cause the model to output the wrong language entirely.
For retrieval tasks, contrastive learning has emerged as a "silver bullet," as it explicitly optimizes the geometry of the embedding space. Furthermore, the survey highlights that massive pre-training from scratch is not always necessary. Modifying existing models like XLM-R through adapters or clever fine-tuning techniques (e.g., xTune or X-MIXUP) often achieves state-of-the-art results with significantly less computational cost.
In terms of specific model performance:
- mDeBERTaV3 is a standout for its size, frequently outperforming larger models in zero-shot transfer.
- LaBSE and mSimCSE remain the standards for retrieval tasks due to their specialized alignment objectives.
- X-MIXUP and VECO show significant gains in "Translate-Train" settings by interpolating between language representations during training.
The Generative Frontier and Future Work
The field is currently shifting from Encoders (understanding-focused) to Decoders/Generative Models (generation-focused). This transition introduces new challenges:
- The Generation Gap: In decoders, alignment must ensure the model respects language boundaries to avoid generating a "word salad" of multiple languages.
- The Curse of Multilinguality: In generative LLMs, adding more languages can degrade performance in the primary language (often English) as they compete for fixed parameter capacity.
- Localization of Alignment: There is ongoing debate regarding where alignment should occur within an LLM—whether in the early layers (acting as an implicit encoder) or the final layers.
- Evaluation Metrics: There is a pressing need for better evaluation methods for multilingual generation that go beyond simple classification accuracy, as current metrics for open-ended generation are often unreliable.
Summary updated