Mixture of LoRA Experts for Continual Information Extraction with LLMs

Zitao Wang, Xinyi Wang, Wei HuEMNLP 2025

TL;DR

This paper introduces MoLE-CIE, a novel approach for continual information extraction (IE) using a mixture of LoRA experts guided by a token-level router. By dynamically selecting specialized LoRA modules for each token and employing a gate reflection mechanism, MoLE-CIE effectively mitigates catastrophic forgetting and enhances knowledge transfer across diverse IE tasks, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Information Extraction (IE) tasks, such as Named Entity Recognition (NER), Relation Extraction (RE), and Event Detection (ED), are crucial for structuring unstructured text. However, in real-world scenarios, new IE tasks and types emerge continuously. Traditional continual learning methods often struggle with "catastrophic forgetting," where learning new information leads to the loss of previously acquired knowledge. Existing approaches for continual IE are often task-specialized, leading to isolated models and hindering knowledge sharing between related tasks. This paper addresses the challenge of building a unified model that can continuously extract emerging information across diverse IE tasks without forgetting past knowledge and while effectively transferring knowledge between tasks.

Methods

The core of the proposed solution is the Mixture of LoRA Experts (MoLE) module integrated into the attention layers of a pre-trained Large Language Model (LLM).

  • Token-Level LoRA Router: Instead of selecting LoRA experts at the sentence level (as in prior work), MoLE-CIE routes each individual token to the most relevant LoRA experts. This allows for finer-grained specialization and better utilization of semantic similarities across tokens from different tasks.
  • Task Experts: To further preserve task-specific knowledge and combat forgetting, dedicated "task experts" are introduced for each IE task. These experts are selected based on the similarity between learned "task keys" and the input token's representation.
  • Gate Reflection Module: This mechanism uses knowledge distillation to prevent the LoRA router and task keys from forgetting. It enforces that the probability distributions predicted by the current router do not deviate significantly from the previous router's predictions, and similarly for task keys. This ensures that the gating mechanisms retain learned knowledge.
  • Experience Replay: A small subset of data from previous tasks is replayed during training to further reinforce learned knowledge and mitigate forgetting.

Datasets

The authors evaluate their model on established benchmarks for three major IE tasks:

  • Event Detection (ED): ACE05-EN+ and MAVEN.
  • Relation Extraction (RE): Few-Rel and TACRED.
  • Named Entity Recognition (NER): OntoNotes-5.0 and i2b2.

The datasets are partitioned into four disjoint splits for each task to simulate a continual learning setting.

Key Insights and Ideas

  • Token-level routing is crucial: Moving from sentence-level to token-level expert selection significantly improves the model's ability to handle diverse IE tasks and retain knowledge, as different parts of a sentence might require different specialized knowledge.
  • Task experts are vital for preventing forgetting: Dedicated experts for each task act as anchors, preserving task-specific knowledge and preventing it from being overwritten by new tasks.
  • Gate reflection is effective for meta-learning: Distilling knowledge for the router and task keys ensures that the model's decision-making mechanisms (how it chooses experts) also learn and adapt without forgetting.
  • LLMs alone aren't enough for continual IE: Even powerful models like GPT-4o struggle in continual IE scenarios, highlighting the continued need for specialized continual learning techniques.

Key Results

  • MoLE-CIE achieves state-of-the-art performance across various continual IE scenarios, outperforming strong competitors like NoRGa by significant margins (e.g., 14.32% and 9.92% accuracy improvement in Sequence 1 and 2, respectively).
  • The model demonstrates superior knowledge transferability, as measured by FWT and BWT metrics, effectively leveraging knowledge from new tasks to improve performance on older ones.
  • MoLE-CIE significantly mitigates catastrophic forgetting, showing the lowest F.Ra scores and maintaining high performance across all splits.
  • Ablation studies confirm the effectiveness of each component: removing LoRA experts, task experts, or the gate reflection mechanism leads to performance degradation.
  • The model's performance is robust across different task sequences and generalizes well to other NLP tasks (SuperNI benchmark).

Limitations and Future Work

  • Limited scope of IE tasks: The current evaluation focuses on NER, RE, and ED. The authors acknowledge that other IE tasks (e.g., sentiment analysis) exist and could be explored.
  • PEFT vs. Full Fine-tuning: The use of Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA might lead to a slight accuracy loss compared to full fine-tuning, though it offers significant computational benefits.
  • Semantic similarity challenges: The model can still struggle with instances where semantics are very similar across tasks (e.g., "Scrutiny" vs. "Scouring"), indicating room for improvement in distinguishing subtle semantic differences.
  • Future work: The authors suggest exploring more sophisticated methods for distinguishing similar semantics and potentially extending the model to a wider range of NLP tasks.

Summary updated

BibTeX
@inproceedings{wang2025mixture,
  author = {Zitao Wang and Xinyi Wang and Wei Hu},
  title = {{Mixture of LoRA Experts for Continual Information Extraction with LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure