Introduction
In the rapidly evolving landscape of natural language processing, Event Detection (ED) serves as a foundational task, identifying event triggers and categorizing them into specific types. However, real-world applications demand systems that can adapt to a continuous stream of new information. This requirement gives rise to Continual Event Detection (CED), where a model must learn a sequence of new tasks without losing the ability to perform previous ones.
The primary obstacle in CED is catastrophic forgetting, a phenomenon where the optimization for new task weights overwrites the parameters essential for old tasks. While existing memory-based strategies—which store and rehearse a subset of past examples—mitigate this, they often suffer from excessive plasticity. This paper introduces LEDOT (Lifelong Event Detection via Optimal Transport), a framework that addresses forgetting by leveraging the latent linguistic knowledge within pre-trained language models (PLMs) as a stabilizing anchor.
Methodology
LEDOT departs from standard fine-tuning practices by arguing that the Masked Language Modeling (MLM) head of a PLM contains vital distributional priors that are typically discarded. The framework integrates three core components to preserve this knowledge while adapting to new tasks.
1. Optimal Transport (OT) Alignment
The central innovation of LEDOT is the use of Optimal Transport to align the model's classification output with the linguistic distribution of the original PLM head. Using the Sinkhorn distance, an computationally efficient variant of OT, the model is forced to maintain a mapping that is consistent with its original pre-trained vocabulary. This ensures that even as the model learns new event types, its internal representations remain grounded in its foundational linguistic understanding.
2. Trainable Cost Matrix
To facilitate OT alignment, the framework utilizes a trainable cost matrix. This matrix defines the "cost" of moving probability mass between the task-specific class distributions and the PLM's word embeddings. The cost is calculated based on the cosine similarity between class embeddings and the embeddings of the PLM's vocabulary, allowing the model to learn semantic relationships between event labels and natural language tokens.
3. Prototype-Based Replay
Rather than storing raw text samples—which can be memory-intensive and prone to overfitting—LEDOT employs a statistical replay strategy. It stores the mean ($\mu$) and diagonal covariance ($\Sigma$) of latent representations for each event class. During the training of subsequent tasks, the model generates synthetic rehearsal samples from these statistical prototypes, providing a generalized memory of past tasks that is more robust than instance-based buffers.
Experimental Results
The authors evaluated LEDOT across two major Event Detection benchmarks (MAVEN and ACE 2005) and one Relation Extraction benchmark (TACRED) to test its generalizability.
Performance on MAVEN and ACE
On the massive MAVEN dataset, LEDOT achieved state-of-the-art (SOTA) results. When integrated with the SharpSeq optimization method, the framework reached a terminal F1 score of 61.49%, significantly outperforming existing baselines. Similar trends were observed on the ACE 2005 benchmark, confirming the model's efficacy on both large-scale and smaller, classic datasets.
Generalization to Relation Extraction
To demonstrate that the benefits of OT alignment extend beyond event detection, the authors applied LEDOT to Continual Relation Extraction (CRE) using the TACRED dataset. The OT-enhanced version of the "RationaleCL" baseline showed consistent accuracy improvements across a 10-task sequence, suggesting that grounding task-specific heads in PLM priors is a broadly applicable strategy for lifelong learning.
Discussion and Ablation Studies
Ablation experiments highlight the critical role of the Optimal Transport component. Removing the OT alignment (LEDOT-OT) resulted in a performance degradation of 2.33% on MAVEN and 3.46% on ACE. These results suggest that the alignment with the PLM's original language modeling head is the primary driver of the model's resistance to catastrophic forgetting.
Furthermore, the study addresses the "wasteful" nature of standard fine-tuning. By treating the pre-trained head as a source of stability rather than a disposable component, LEDOT effectively balances the trade-off between plasticity (learning new things) and stability (retaining old things).
Limitations and Future Directions
While LEDOT sets a new benchmark for CED, certain limitations remain:
- Architecture Specificity: The current research focuses on encoder-based models like BERT. While these remain highly effective for information extraction, future work is needed to adapt LEDOT for decoder-only Large Language Models (LLMs).
- Computational Complexity: Although the Sinkhorn algorithm is efficient, OT calculations over large vocabularies introduce overhead. The authors currently mitigate this by sub-sampling "verb" tokens, but further optimization could enhance scalability.
- Task Expansion: Future iterations of the framework may explore more complex extraction tasks, such as event-event relation extraction and document-level event synthesis.
Conclusion
LEDOT demonstrates that catastrophic forgetting in lifelong learning can be significantly mitigated by anchoring new knowledge to the linguistic priors of pre-trained models. Through the combination of Optimal Transport alignment and statistical prototype replay, the framework provides a robust and scalable solution for maintaining high performance in dynamic, real-world information environments.
Summary updated