Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

Zi-Yi Dou, Keyi Yu, Antonios Anastasopoulos

Abstract

Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks. Previously, researchers have proposed to use language model pre-training and multi-task learning to learn robust representations. However, these methods can achieve sub-optimal performance in low-resource scenarios. Inspired by the recent success of optimization-based meta-learning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong baselines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively.

Introduction

In Natural Language Processing (NLP), "high-resource" tasks with millions of labeled examples often dominate research. However, many practical NLU problems are "low-resource," characterized by extremely limited labeled data. Standard approaches like Multi-Task Learning (MTL) attempt to leverage knowledge from high-resource tasks to improve performance on low-resource ones. However, MTL often prioritizes high-resource tasks, resulting in sub-optimal representations for smaller datasets. This research investigates meta-learning, or "learning to learn," to find a model initialization specifically optimized for rapid adaptation to data-sparse environments.

Methodology

The study implements three optimization-based meta-learning algorithms on a BERT-Base architecture:

  1. MAML (Model-Agnostic Meta-Learning): Searches for a parameter initialization that can reach an optimal solution for a new task in minimal gradient steps via second-order meta-optimization.
  2. FOMAML (First-Order MAML): A computationally efficient variant that ignores second-order derivatives.
  3. Reptile: A first-order algorithm that moves global weights toward task-specific weights after several gradient descent steps.
Conceptual comparison of Multi-task learning and Meta-learning.

Unlike MTL, which seeks a single joint solution for all tasks, meta-learning aims for a "versatile initialization" or launchpad. As illustrated in Figure 1, this starting point is positioned to be closer to the optimal solution for any individual task within the distribution, facilitating faster convergence.

Experimental Setup

The models were evaluated using the GLUE benchmark. High-resource tasks (SST-2, QQP, MNLI, QNLI) served as auxiliary tasks for meta-learning, while low-resource tasks (CoLA, MRPC, STS-B, RTE) were used for final evaluation. Additionally, the SciTail dataset was used to test cross-domain transfer learning and data efficiency.

Results and Discussion

Meta-learning models consistently outperformed both standard BERT-Base and the Multi-Task Deep Neural Network (MT-DNN) baseline on low-resource tasks. Notably, Reptile achieved superior results on MRPC (90.2) and RTE (77.0), while MAML led on CoLA (53.4).

Performance comparison across different percentages of training data on the SciTail dataset.

A key finding is the extreme data efficiency of meta-learned models. As shown in Figure 2, when training data is reduced to 0.1%, the Reptile-based model maintains significantly higher accuracy (~81%) compared to the standard BERT baseline (~55%). Probing analysis further suggests that meta-learned models retain more general linguistic information in lower layers, avoiding the task-specific overfitting common in standard MTL.

Conclusion

Optimization-based meta-learning, particularly the Reptile algorithm, offers a robust framework for low-resource NLU. By focusing on versatile initialization rather than joint optimization, these methods enable rapid and effective adaptation to new tasks with minimal data. Future work includes extending these algorithms to cross-lingual tasks and exploring more sophisticated hybrid meta-learning strategies.

Summary updated

Abstract

Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks. Previously, researchers have proposed to use language model pre-training and multi-task learning to learn robust representations. However, these methods can achieve sub-optimal performance in low-resource scenarios. Inspired by the recent success of optimization-based meta-learning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong baselines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively.
BibTeX
@inproceedings{dou2019investigating,
  author = {Zi-Yi Dou and Keyi Yu and Antonios Anastasopoulos},
  title = {{Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Oe0YbwIskAl},
}
Figure