Introduction
Information Extraction (IE) is a fundamental task in natural language processing, aimed at transforming unstructured text into structured representations such as entities, relations, and events. Recent advancements have shifted toward Universal Information Extraction (UIE) models capable of handling multiple tasks simultaneously. However, real-world applications often present data in a continuous stream of new domains and tasks, necessitating a lifelong learning approach.
Problem Statement: Lifelong Information Extraction
Standard UIE models face two primary challenges in a lifelong learning scenario:
- Catastrophic Forgetting (CF): When a model is trained on a new task, it often loses the ability to perform previously learned tasks.
- Knowledge Transfer (KT): A robust lifelong learner should leverage prior knowledge to improve performance on subsequent tasks without requiring excessive memory for old data.
Methodology: Lottery Prompt Tuning (LPT)
To address these challenges, we propose Lottery Prompt Tuning (LPT). This method keeps a large Pre-trained Language Model (PLM) frozen and only optimizes a small set of "soft prompt" vectors. LPT draws inspiration from the Lottery Ticket Hypothesis, which posits that large neural networks contain sparse sub-networks capable of matching the performance of the full model.
Task-Aware Parameter Isolation
LPT applies this hypothesis to prompt tuning by learning a task-aware binary mask for each new task. This mask identifies a specific subset—a "lottery ticket"—of the shared prompt parameters. Once a parameter is assigned to a task, it is frozen for all future tasks, physically preventing the overwriting of old knowledge and eliminating catastrophic forgetting.
Online Pruning and Selection
Instead of iterative pruning, LPT utilizes a "Straight-Through Estimator" to learn importance scores and masks concurrently during training. For novel or unseen tasks, the model employs a perplexity-based or gradient-based selection method to identify the most suitable mask from its library of previously learned tasks, facilitating zero-shot transfer.
Experimental Evaluation
The LPT framework was evaluated across a sequence of 13 IE datasets, covering four major extraction types: entity, relation, event, and aspect-based sentiment analysis. This benchmark includes diverse datasets such as ACE05, CoNLL04, and SemEval, testing the model's ability to scale across varied linguistic domains.
Key Results and Insights
LPT demonstrates significant improvements over existing lifelong learning baselines such as EWC, ER, and L2P:
- Superior Performance: LPT achieved an Average F1 score of 76.914 across 13 tasks, a 3% improvement over the previous state-of-the-art (L2P).
- Zero Forgetting: By design, LPT achieved a Backward Transfer (BWT) score of 0, indicating complete preservation of prior knowledge through parameter isolation.
- Strong Generalization: In zero-shot settings, LPT significantly outperformed baselines (e.g., 32.69 F1 vs. 20.88 for L2P), validating the effectiveness of its mask-selection strategy for knowledge transfer.
- Efficiency: The method is highly parameter-efficient, tuning only 0.097% of the model's parameters per task, making it suitable for deployment in resource-constrained environments.
Conclusion
Lottery Prompt Tuning (LPT) provides a scalable and efficient solution for lifelong information extraction. By isolating task-specific "lottery tickets" within a shared prompt space, LPT effectively balances the need for knowledge transfer with the requirement for zero forgetting. Future work will explore adaptive sparsity ratios and more complex prompt ensembling techniques to further enhance performance.
Reference: Liang, Z., et al. (2023). Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023).
Summary updated