Generalization Measures for Zero-Shot Cross-Lingual Transfer

Saksham Bassi, Duygu Ataman, Kyunghyun Cho

TL;DR

This paper explores how the "geometry" of a model's loss landscape can predict its ability to perform zero-shot cross-lingual transfer. The authors demonstrate that models with "flatter" loss landscapes and higher prediction margins generalize significantly better to unseen languages, and they introduce a novel, stable algorithm to measure this sharpness 5–10x faster than previous methods.

What Problem Does This Paper Address?

In natural language processing, we often train a model on a high-resource language (like English) and hope it "just works" on a low-resource language (like Swahili or Urdu) without further training—a feat known as zero-shot cross-lingual transfer.

The problem is that we usually don't know how well a model will generalize until we actually test it on those target languages. However, for many languages, we lack the gold-standard evaluation data to do this. Traditional metrics tell us how well a model learned the training data, but they are poor predictors of how it will handle "unseen" linguistic distributions. This paper seeks a "generalization measure"—a way to look at the model's internal state after training on English and predict its success across the globe.

Methods

The researchers focused on two primary indicators of generalization:

  1. Margin: A measure of how "confident" a model is. It’s the difference between the probability assigned to the correct label and the next most likely label.
  2. Sharpness: This describes the "ruggedness" of the loss landscape. If you nudge the model's weights slightly, does the error (loss) explode (sharp) or stay roughly the same (flat)?

To test these, they compared several optimization strategies designed to find flatter minima:

  • SAM (Sharpness-Aware Minimization): Explicitly seeks out regions of the weight space where the loss is uniformly low.
  • FIM (Fisher Information Matrix): A regularization technique that penalizes high local curvature in the loss landscape.
  • MVR (Multi-view Subword Regularization): A technique that forces the model to be consistent even when the input text is segmented into subwords in different ways.

The Novelty: The authors proposed a new Difference-based Sharpness algorithm ($\phi_{difference}$). Previous methods for calculating sharpness were often unstable or computationally expensive (requiring binary searches for optimal radii). Their approach uses a simpler, gradient-based perturbation that is more stable and significantly faster to compute.

Datasets

The study utilized two major multilingual benchmarks:

  • XNLI (Cross-lingual Natural Language Inference): Models were trained on English and evaluated on 14 other languages, including Arabic, Greek, Hindi, Swahili, and Vietnamese.
  • PAWS-X (Paraphrase Adversaries from Word Scrambling): A dataset for identifying paraphrases across 7 languages (German, English, Spanish, French, Japanese, Korean, and Chinese).

They tested these measures across different architectures, primarily mBERT (multilingual BERT), RoBERTa, XLM-R, and mT5.

Key Insights and Ideas

  • Flatness Equals Flexibility: The central "aha moment" is the confirmation of the flatness hypothesis in a cross-lingual context. A model that sits in a wide, flat "valley" of low loss is less likely to be tripped up by the structural differences of a new language compared to a model stuck in a narrow, "sharp" pit.
  • Margin is a Powerhouse Predictor: Across almost all experiments, the "Margin" (prediction confidence) showed an incredibly high correlation (often >0.9) with actual test accuracy on unseen languages. If a model is very sure of its answers in English, it’s a very good sign it will be right in Hindi.
  • Optimization Matters: Standard training (like AdamW) often falls into sharp minima. Using min-max optimizers like SAM or FIM consistently produced flatter landscapes and, consequently, better zero-shot performance.
  • Geometry over Distance: Interestingly, the authors found that the distance the weights moved from their starting point (Frobenius distance) was not a reliable predictor of generalization, debunking some previous theories in this specific cross-lingual setting.

Key Results

  • Correlation Strength: On the XNLI dataset, the correlation between Margin and Test Accuracy reached as high as 0.954 for mBERT trained with Fisher Penalty.
  • Sharpness Correlation: Sharpness showed a strong negative correlation (up to -0.845), meaning as the landscape gets sharper, performance on other languages reliably drops.
  • Stability: The proposed $\phi_{difference}$ algorithm provided more consistent results than previous sharpness formulas, which often hit "extreme points" during calculation and failed to provide meaningful gradients.

Limitations and Future Work

While the results are promising, the authors note a few hurdles:

  • Heuristic Tuning: Their new sharpness algorithm still requires manual tuning of certain "hyper-parameters" (like the noise scale) to get the best results.
  • Classification Only: The "Margin" metric is currently only defined for classification tasks. It’s not yet clear how to apply this logic to generative models (like ChatGPT-style LLMs) where the "label" isn't a simple choice between three categories.
  • Scope: The study focused on zero-shot transfer. Future work could explore if these measures also predict performance after "few-shot" fine-tuning in the target language.

Summary updated

Abstract

Building robust and reliable machine learning systems requires models with the capacity to generalize their knowledge to interpret unseen inputs with different characteristics.Traditional language model evaluation tasks lack informative metrics about model generalization, and their applicability in new settings is often measured using task and language-specific downstream performance, which is lacking in many languages and tasks.To address this gap, we explore a set of efficient and reliable measures that could aid in computing more information related to the generalization capability of language models, particularly in cross-lingual zero-shot settings.Our central hypothesis is that the sharpness of a model's loss landscape, i.e., the representation of loss values over its weight space, can indicate its generalization potential, with a flatter landscape suggesting better generalization.We propose a novel and stable algorithm to reliably compute the sharpness of a model optimum, and demonstrate its correlation with successful cross-lingual transfer. 1
BibTeX
@misc{bassi2024generalization,
  author = {Saksham Bassi and Duygu Ataman and Kyunghyun Cho},
  title = {{Generalization Measures for Zero-Shot Cross-Lingual Transfer}},
  year = {2024},
  eprint = {2404.15928},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.15928},
}
Figure