Understanding Cross-Lingual Alignment - A Survey

Katharina Hämmerl, Jindrich Libovický, Alexander Fraser

Abstract: Cross-lingual alignment, the meaningful similarity of representations across languages in multilingual language models, has been an active field of research in recent years. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field. We present different understandings of cross-lingual alignment and their limitations. We provide a qualitative summary of results from a number of surveyed papers. Finally, we discuss how these insights may be applied not only to encoder models, where this topic has been heavily studied, but also to encoder-decoder or even decoder-only models, and argue that an effective trade-off between language-neutral and language-specific information is key.

Introduction

Multilingual Large Language Models (MLLMs) have demonstrated a remarkable capacity for "zero-shot cross-lingual transfer," where a model trained on a task in one language (e.g., English) can perform it in another (e.g., Swahili) without further training. This capability is fundamentally rooted in cross-lingual alignment: the degree to which a model's internal representations of different languages are mapped into a shared semantic space.

Despite its importance, the field has lacked a unified definition of alignment. Current research oscillates between two primary views: View I focuses on geometric similarity (e.g., high cosine similarity between translations), while View II emphasizes the ability of a task-specific "head" to extract patterns regardless of the input language. As the field shifts from discriminative encoders to generative decoders, understanding this alignment becomes critical, as excessive alignment may inadvertently strip away necessary language-specific nuances.

Taxonomy of Alignment Strategies

The survey organizes the vast landscape of alignment techniques into a structured taxonomy based on training objectives and data requirements. These methods range from those that refine existing models to those that build aligned representations from scratch.

Taxonomy of cross-lingual alignment methods categorized by training objectives and initialization.

Key categories include:

  • Parallel Data Objectives: Utilizing bitexts at the sentence level (e.g., LASER) or word level (e.g., XLM-Align) to explicitly force representations together.
  • Contrastive Learning: Learning to pull translation pairs closer in the embedding space while pushing non-translations apart (e.g., InfoXLM, LaBSE).
  • Modified Pre-training: Altering core architectures or loss functions, such as employing ELECTRA-style replaced token detection (e.g., mDeBERTaV3).
  • Adapters and Transformations: Using small, language-specific modules (e.g., MAD-X) or post-hoc mathematical adjustments like rotating the embedding space (e.g., RotateAlign) to bridge language gaps without full retraining.

Evaluation Benchmarks

To measure the efficacy of these alignment strategies, the research community relies on several standard benchmarks:

  • XNLI & PAWS-X: Standard tasks for cross-lingual Natural Language Inference and paraphrase identification.
  • MLQA & UD-POS: Benchmarks for multilingual Question Answering and Part-of-Speech tagging.
  • Tatoeba & BUCC: Specialized datasets for evaluating sentence retrieval and bitext mining capabilities.

Theoretical Frameworks and Insights

A core contribution of this survey is the Two-View Framework. While View I (geometric similarity) is easier to quantify, the authors argue that View II (subspace transfer) better explains model performance. A model may exhibit "messy" global alignment but still achieve perfect transfer if task-relevant information resides in a language-neutral subspace.

Furthermore, the survey highlights that "perfect" alignment is not always the optimal goal. While vital for retrieval tasks, strong alignment can be detrimental to generative tasks, where maintaining language-specific style and grammar is essential. This tension is part of the "Curse of Multilinguality," where increasing the number of supported languages can dilute per-language performance.

Performance Analysis

  • Encoder Models: mDeBERTaV3 stands out at smaller scales due to its efficient pre-training, while XY-LENT leads among billion-parameter models.
  • Retrieval Excellence: Models like LaBSE and mSimCSE, which utilize contrastive learning, remain the gold standard for achieving the strong geometric alignment required for sentence retrieval.
  • Data Efficiency: The inclusion of even minimal parallel data during fine-tuning (translate-train) consistently yields superior results compared to pure zero-shot transfer.

Future Directions

The survey identifies several critical gaps for future research. First, the field must move beyond the "English Pivot" toward true many-to-many alignment. Second, there is a pressing need for better evaluation metrics that assess how alignment impacts the fluency and factuality of generative models, rather than just classification accuracy. Finally, as decoder-only models like Llama become dominant, identifying and targeting the "implicit encoders" within their early layers represents a major frontier for cross-lingual research.

Summary updated

Abstract

Cross-lingual alignment, the meaningful similarity of representations across languages in multilingual language models, has been an active field of research in recent years. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field. We present different understandings of cross-lingual alignment and their limitations. We provide a qualitative summary of results from a number of surveyed papers. Finally, we discuss how these insights may be applied not only to encoder models, where this topic has been heavily studied, but also to encoder-decoder or even decoder-only models, and argue that an effective trade-off between language-neutral and language-specific information is key.
BibTeX
@inproceedings{hammerl2024understanding,
  author = {Katharina Hämmerl and Jindrich Libovický and Alexander Fraser},
  title = {{Understanding Cross-Lingual Alignment - A Survey}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=xmXbNC0Lrq},
}
Figure