Abstract
Zi-Yi Dou, Keyi Yu, Antonios Anastasopoulos. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Large-scale pre-trained language models like BERT have revolutionized Natural Language Processing (NLP). However, these models often struggle when applied to new tasks with limited training data. While Multi-Task Learning (MTL) is a common strategy to address data scarcity by sharing knowledge across tasks, it frequently suffers from a bias toward high-resource tasks whose gradients dominate the training process. This paper investigates whether optimization-based meta-learning can provide a more robust solution by specifically optimizing a model's ability to adapt quickly to new, low-resource tasks.
Methodology
The authors explore the application of three optimization-based meta-learning algorithms to a BERT-Base architecture: Model-Agnostic Meta-Learning (MAML), First-Order MAML (FOMAML), and Reptile. Unlike standard MTL, which seeks a single parameter set that performs well across all tasks simultaneously, meta-learning aims to find an initialization $\theta$ that serves as an ideal starting point for rapid adaptation to specific target tasks.
The study utilizes the GLUE benchmark, designating high-resource tasks (SST-2, QQP, MNLI, QNLI) as auxiliary tasks for meta-training. The resulting model is then evaluated on low-resource target tasks (CoLA, MRPC, STS-B, RTE). To further test generalization, the researchers evaluate the model on SciTail, a science-domain entailment dataset not seen during the meta-learning phase.
Key Insights
- Adaptability as an Objective: Meta-learning effectively finds a "launchpad" initialization. This allows the model to "sprint" toward a task-specific solution with minimal gradient steps.
- Algorithm Performance: Reptile emerged as a particularly effective algorithm, offering a balance of stability and computational efficiency that often outperformed the more complex MAML.
- Hyperparameter Sensitivity: The "inner loop"—the number of gradient steps taken during meta-training—is critical. The authors found that 5 steps provided the optimal balance between joint training and uninformative gradients.
- Task Sampling: Sampling tasks proportional to their dataset size (PPS) proved superior to uniform sampling, likely due to the increased stability provided by larger datasets.
Results
Meta-learning approaches consistently outperformed both standard BERT and MT-DNN baselines on low-resource GLUE tasks. For instance, on the RTE task, Reptile achieved 77.0% accuracy, a significant improvement over BERT's 66.4%.
The most notable gains were observed in few-shot transfer scenarios. On the SciTail dataset, with only 0.1% of training data (approx. 23 examples), the Reptile-initialized model reached nearly 80% accuracy, whereas the standard BERT model performed near random chance at 55%.
Conclusion
Optimization-based meta-learning, particularly the Reptile algorithm, offers a powerful framework for improving the adaptability of large-scale language models to low-resource NLU tasks. While it introduces additional computational overhead compared to standard fine-tuning, the gains in few-shot performance and cross-domain generalization are substantial. Future research may explore these techniques in multilingual contexts or integrate more advanced meta-learning strategies.
Summary updated