Minimax and Neyman-Pearson Meta-Learning for Outlier Languages

Edoardo Maria Ponti, Rahul Aralikatte, Disha Shrivastava, Siva Reddy, Anders Søgaard

TL;DR

This paper tackles the challenge of outlier languages in meta-learning for NLP, where standard methods struggle with linguistic diversity. They introduce Minimax MAML and Neyman-Pearson MAML, novel criteria that are more robust to under-represented languages by optimizing for worst-case scenarios or imposing risk constraints. These new approaches, coupled with an adaptive optimizer, show improved performance on low-resource languages for tasks like POS tagging and question answering.

What Problem Does This Paper Address?

Cross-lingual transfer learning in NLP aims to leverage knowledge from high-resource languages to improve performance on low-resource ones. Model-Agnostic Meta-Learning (MAML) has emerged as a powerful paradigm for this, enabling models to adapt quickly with minimal data. However, MAML typically assumes that training and evaluation languages are independent and identically distributed (i.i.d.). This assumption is often violated in practice because existing datasets tend to be skewed towards a few language families and geographical regions. Consequently, many of the world's languages are "outliers" – they don't fit the patterns learned from the training data, leading to poor generalization. This paper addresses this critical gap by proposing meta-learning objectives that are explicitly designed to be robust to such out-of-distribution language scenarios.

Methods

The core of this work lies in re-framing MAML from a decision-theoretic perspective. Instead of minimizing the average risk across training languages (the standard Bayes criterion), the authors propose two alternative criteria:

  • Minimax MAML: This approach aims to be robust to the worst-case scenario by minimizing the maximum risk across languages. It learns an adversarial distribution over languages to identify and mitigate the highest-risk scenarios.
  • Neyman-Pearson MAML: This criterion imposes constraints on the risk for a subset of languages, ensuring that the risk for any given language does not exceed a predetermined threshold. This is achieved using Lagrange multipliers, effectively creating a constrained optimization problem.

Both Minimax and Neyman-Pearson MAML are formulated as two-player games, where one player (the model) minimizes loss and the other (nature or the adversarial distribution) maximizes it. To handle the non-convexity of deep neural networks, they propose an Adaptive Symplectic Gradient Adjustment (ASGA) optimizer. ASGA builds upon existing Symplectic Gradient Adjustment (SGA) methods by incorporating adaptive learning rates and momentum, inspired by Adam, to better handle these competitive game dynamics.

Datasets

The experiments were conducted on two widely used NLP benchmarks:

  • Universal Dependencies (UD): Used for Part-of-Speech (POS) tagging. This dataset provides a broad coverage of languages.
  • TyDiQA: Used for question answering.

The authors carefully partition languages for evaluation, holding out language isolates and languages with few family members to simulate realistic low-resource scenarios. This ensures that the evaluation sets truly represent "outlier" languages relative to the training data.

Key Insights and Ideas

  • The i.i.d. assumption is a major bottleneck for cross-lingual NLP: The paper strongly argues that the standard i.i.d. assumption in MAML is fundamentally at odds with the skewed distribution of languages in real-world NLP datasets.
  • Decision theory offers a powerful lens for robust meta-learning: By re-interpreting MAML through a decision-theoretic framework, the authors unlock new criteria that directly address out-of-distribution challenges.
  • Minimax and Neyman-Pearson criteria offer complementary robustness: Minimax MAML focuses on the absolute worst-case, while Neyman-Pearson MAML provides a more controlled approach with risk bounds. Both are shown to be beneficial.
  • Adaptive optimization is crucial for competitive games: The proposed ASGA optimizer is essential for effectively training models with these adversarial criteria, outperforming simpler gradient-based methods in certain scenarios.
  • Outlier languages benefit most: The most significant gains for the proposed methods are observed on the most disadvantaged languages, highlighting their effectiveness in improving fairness and robustness.

Key Results

The paper reports promising results across POS tagging and question answering:

  • POS Tagging: Minimax MAML (MM+) and Neyman-Pearson MAML (NP+) consistently outperform vanilla MAML (B) and joint multi-source transfer (J) baselines, especially in few-shot settings. For example, at k=0 (zero-shot), MM+ shows a 0.94 point gain in F1 score over B. Crucially, the minimum F1 scores across languages are significantly uplifted by NP and NP+, showing substantial gains (7-12 F1 points) over baselines, indicating improved performance on the most challenging languages.
  • Question Answering (QA): While the results are more mixed, MM+ and NP+ still show improvements over baselines, particularly in the Exact Match metric. The authors speculate that the larger language coverage in POS tagging might contribute to better generalization for meta-learning compared to QA.
  • Optimizer Performance: ASGA generally shows favorable results for POS tagging compared to Adam-optimized MAML, though the trend is reversed for QA, suggesting further investigation into optimizer choices.

Limitations and Future Work

  • Hyperparameter Sensitivity: The authors note that they did not perform extensive hyperparameter searches for the adversarial parameters (τ and λ), the threshold (r), or differential learning rates. This suggests that further tuning could potentially yield even better results.
  • Mixed Results in QA: The performance gains for QA were less consistent than for POS tagging. The authors attribute this to the differences in language coverage and distribution between the UD and TyDiQA datasets.
  • Generalizability: While the paper demonstrates effectiveness in NLP tasks, the authors suggest that the proposed criteria and optimizers are more general and could be applied to other machine learning domains, such as regression (as briefly illustrated in Appendix C).
  • Future Investigations: Further research is needed to explore modifications to the Symplectic Gradient Adjustment, particularly concerning its performance on QA tasks, and to conduct more extensive hyperparameter tuning.

Summary updated

Abstract

Model-agnostic meta-learning (MAML) has been recently put forth as a strategy to learn resource-poor languages in a sample-efficient fashion. Nevertheless, the properties of these languages are often not well represented by those available during training. Hence, we argue that the i.i.d. assumption ingrained in MAML makes it ill-suited for cross-lingual NLP. In fact, under a decision-theoretic framework, MAML can be interpreted as minimising the expected risk across training languages (with a uniform prior), which is known as Bayes criterion. To increase its robustness to outlier languages, we create two variants of MAML based on alternative criteria: Minimax MAML reduces the maximum risk across languages, while Neyman-Pearson MAML constrains the risk in each language to a maximum threshold. Both criteria constitute fully differentiable two-player games. In light of this, we propose a new adaptive optimiser solving for a local approximation to their Nash equilibrium. We evaluate both model variants on two popular NLP tasks, part-of-speech tagging and question answering. We report gains for their average and minimum performance across low-resource languages in zero- and few-shot settings, compared to joint multi-source transfer and vanilla MAML.
BibTeX
@inproceedings{ponti2021minimax,
  author = {Edoardo Maria Ponti and Rahul Aralikatte and Disha Shrivastava and Siva Reddy and Anders Søgaard},
  title = {{Minimax and Neyman-Pearson Meta-Learning for Outlier Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.106},
}
Figure