Meta-Learning for Few-Shot Named Entity Recognition

Cyprien de Lichy, Hadrien Glaude, William CampbellACL 2021

TL;DR

This paper tackles the challenge of Named Entity Recognition (NER) when only a handful of labeled examples are available (few-shot NER). They propose a novel task generation scheme and adapt meta-learning algorithms (Prototypical Networks and Reptile) to achieve state-of-the-art performance, outperforming fine-tuned BERT baselines.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is crucial for extracting structured information from text, powering applications like virtual assistants and document parsing. However, training effective NER models typically requires large amounts of labeled data. Acquiring this data is expensive and time-consuming. While transfer learning, particularly fine-tuning large language models like BERT, has improved performance, it still demands a significant number of task-specific examples. This becomes a bottleneck for personalized models or scenarios with very limited data budgets. This paper addresses this "few-shot NER" problem, where models must generalize to new entity types with only a few labeled examples.

Methods

The authors propose a novel approach to few-shot NER by leveraging meta-learning. Their core contributions include:

  • Novel Few-Shot NER Definition: They define few-shot NER as a task where models must learn to identify a set of target entity types given only a small "support set" of labeled utterances for each type.
  • Task Generation Scheme: They introduce a method to convert classical NER datasets into few-shot tasks for both meta-training and meta-testing. This involves partitioning entity types and utterances, ensuring that no test entities are seen during training to prevent data leakage.
  • Meta-Learning Algorithms:
    • Prototypical Networks (ProtoNet): They adapt Prototypical Networks, a metric-based meta-learning approach, to handle structured prediction tasks like NER. This involves embedding words in a context-aware manner and computing prototypes for each entity type. They also propose combining this with Conditional Random Fields (CRF) for better sequence labeling.
  • Reptile: They apply Reptile, an optimization-based meta-learning algorithm, to learn a good initialization for the NER model. This allows for faster adaptation to new tasks with few examples.
  • Proto-Reptile: They propose a novel combination of Prototypical Networks and Reptile, aiming to leverage the strengths of both metric-based and optimization-based meta-learning.
  • BERT Integration: They utilize pre-trained BERT models for word embeddings, fine-tuning them as part of their meta-learning process.

Datasets

The authors evaluated their methods on three public datasets:

  • SNIPS: A dataset for spoken language understanding.
  • Task Oriented Parsing (TOP): A dataset for dialogue state tracking.
  • Google Schema-Guided Dialogue State Tracking (DSTC8): Another dialogue state tracking dataset.

They did not adhere to pre-defined train/validation/test splits but used their own task-based generation procedure (described in Section 3.4) to create diverse few-shot tasks.

Key Insights and Ideas

  • Meta-learning is crucial for low-data regimes: The paper strongly suggests that meta-learning approaches significantly outperform standard fine-tuning baselines when data is scarce, demonstrating the power of learning to learn.
  • CRF decoder is beneficial: Combining Prototypical Networks with a CRF decoder leads to substantial improvements, especially on datasets with higher diversity like DSTC8. This highlights the importance of modeling label dependencies in sequence labeling.
  • Fine-tuning Prototypical Networks is key: For textual applications built on pre-trained language models, fine-tuning Prototypical Networks at meta-test time (using support examples) is highly beneficial, unlike in some image recognition tasks.
  • Task diversity drives meta-learning success: The performance of meta-learning approaches is significantly boosted by the diversity of tasks encountered during meta-training. This suggests that richer task distributions lead to a more effective learned inductive bias.
  • Robustness to N/K mismatches: The proposed Proto-Reptile approach shows robustness to variations in the number of shots (K) and ways (N) between meta-training and meta-testing, which is a valuable characteristic for real-world applications.

Key Results

  • State-of-the-art performance: The proposed Proto-Reptile approach with a CRF decoder achieved the best results across the evaluated datasets, significantly outperforming a fine-tuned BERT baseline.
  • Proto-Reptile + CRF: For example, on DSTC8, Proto-Reptile + CRF achieved a Micro F1 of 83.83 ± 4.13, compared to the baseline's 47.08 ± 7.02.
  • ProtoNet + CRF: ProtoNet + CRF also showed strong performance, achieving 89.67 ± 0.63 on SNIPS.
  • Fine-tuning gains: Fine-tuning the meta-learned models at meta-test time provided substantial gains, especially for the ProtoNet and Proto-Reptile approaches.
  • ProtoNet and Proto-Reptile (without fine-tuning): These versions still performed competitively, but fine-tuning offered a significant boost.

Limitations and Future Work

  • "Other" class handling: The paper acknowledges that the handling of the "other" (background) class could be improved. They suggest that this class is partially shared across tasks and could be leveraged more effectively.
  • Integration of unseen entities: Future work could explore integrating few-shot learning for models that need to perform well on tasks with both seen and unseen entity types during meta-training.
  • Optimization-based meta-learning capacity: The authors speculate that other optimization-based meta-learning methods with larger capacity external neural networks might be more efficient than relying solely on initialization weights for learning inductive bias.
  • Computational cost: While not explicitly stated as a limitation, hyper-parameter optimization (HPO) using Bayesian optimization was computationally intensive.

Summary updated

BibTeX
@inproceedings{lichy2021meta,
  author = {Cyprien de Lichy and Hadrien Glaude and William Campbell},
  title = {{Meta-Learning for Few-Shot Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure