FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction

Minh Van Nguyen, Nghia Ngo, Bonan Min, Thien NguyenNAACL 2022

TL;DR

This paper introduces FAMIE, a novel active learning (AL) framework designed to accelerate multilingual information extraction. By employing a lightweight "proxy" model for rapid data selection and a knowledge distillation mechanism to keep it aligned with a larger, more powerful model, FAMIE significantly reduces annotator waiting times without sacrificing performance. This makes AL more practical and efficient for real-world applications.

What Problem Does This Paper Address?

Information Extraction (IE) systems are crucial for structuring data from text, but building high-quality IE models typically requires vast amounts of labeled data. Active learning (AL) offers a solution by intelligently selecting the most informative data points for human annotation, thereby reducing labeling effort. However, existing AL frameworks for sequence labeling tasks (like Named Entity Recognition and Event Detection) suffer from a critical bottleneck: the time it takes to train the main model and select new data samples at each AL iteration. This long "idle time" for annotators can lead to decreased productivity, engagement, and ultimately, fewer annotated examples within a given budget. This problem is exacerbated by the trend of using large, powerful pre-trained language models (like BERT), which are computationally expensive to train.

Methods

FAMIE tackles the AL bottleneck with a multi-pronged approach:

  • Proxy Model for Fast Data Selection: Instead of training the full, large-scale model for data selection at each AL iteration, FAMIE uses a smaller, "proxy" model. This proxy model is trained much faster, drastically reducing the time spent on data selection.
  • Knowledge Distillation for Alignment: A key challenge is ensuring that the examples selected by the fast proxy model are also optimal for the main, powerful model. FAMIE addresses this with a novel knowledge distillation mechanism. This process synchronizes the proxy model with the main model, ensuring the selected examples are relevant and informative for the larger model.
  • Parallel Training: To further optimize annotator time, FAMIE proposes training the main model concurrently with the human annotation process. This means the computing resources are utilized during the annotation phase, rather than sitting idle.
  • Adapter-based Fine-tuning: For efficiency, FAMIE incorporates lightweight adapter networks. These are small feed-forward networks injected into the pre-trained language model (XLM-Roberta in this case). During training, only the adapters are updated, significantly reducing the number of trainable parameters and thus training time, while often maintaining or even improving performance, especially in low-resource scenarios.
  • Multilingual Support: FAMIE leverages the Trankit toolkit, enabling it to support sequence labeling tasks across 56 languages.

Datasets

The paper evaluates FAMIE on the following datasets for Named Entity Recognition (NER) and Event Detection (ED):

  • CoNLL03-English: For NER.
  • CoNLL02-Spanish: For NER.
  • ACE-English: For ED.
  • ACE-Chinese: For ED.

The authors follow standard data splits for train/dev/test portions. For AL experiments, the unlabeled pool is created by discarding labels from the original training data.

Key Insights and Ideas

  • The "Idle Time" Problem is Real and Costly: The paper strongly emphasizes that the waiting time for annotators in traditional AL is a significant practical barrier, impacting productivity and engagement.
  • Proxy Models are Viable for Speed: Using a smaller proxy model for data selection is an effective strategy to dramatically reduce computation time, making AL more responsive.
  • Distillation is Key to Bridging the Gap: The success of the proxy model hinges on its alignment with the main model. Knowledge distillation is presented as a crucial technique to achieve this alignment, preventing performance degradation.
  • Parallelization is a Game-Changer: Training the main model during annotation time is a simple yet powerful idea that fully exploits available computational resources and human effort.
  • Adapters Offer Efficiency Without Sacrifice: Adapter-based fine-tuning provides a significant speedup in training the main model without a substantial drop in performance, and can even be beneficial in certain scenarios.

Key Results

  • Significant Time Reduction: FAMIE reduces annotator waiting time to approximately 3.4 minutes per AL iteration, which is over 10 times faster than traditional AL approaches (e.g., "Large" baseline, which took over 30 minutes per iteration after 8 rounds).
  • Competitive Performance: Despite the speed improvements, FAMIE achieves performance only marginally below that of a full-scale AL baseline ("Large"), demonstrating that the proxy model and distillation are effective.
  • Knowledge Distillation is Crucial: Removing the distillation component ("FAMIE-AD") resulted in performance that was similar to or worse than random selection, highlighting the importance of aligning the proxy and main models.
  • Adapters Boost Efficiency and Performance: Using adapters ("FAMIE-A" and "FAMIE-AD") further reduced training time and, in some cases, improved performance.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while FAMIE significantly reduces waiting time, the main model still requires training, which can take around 32 minutes per iteration on average. However, this is framed as manageable within the annotation time.
  • The performance of FAMIE-AD (without distillation) suggests that the alignment mechanism is critical; without it, the proxy model's selections might not be optimal for the main model.
  • Future Work:
    • The authors plan to extend FAMIE to cover other NLP tasks beyond sequence labeling, given its task-agnostic nature.

Summary updated

BibTeX
@inproceedings{nguyen2022famie,
  author = {Minh Van Nguyen and Nghia Ngo and Bonan Min and Thien Nguyen},
  title = {{FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure