Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning

Zujie Liang, Feng Wei, Yin Jie, Yuxi Qian, Zhenghong Hao, Bing HanACL 2023 · DOI

Introduction

Information Extraction (IE) is a fundamental task in natural language processing, aimed at transforming unstructured text into structured representations such as entities, relations, and events. Recent advancements have shifted toward Universal Information Extraction (UIE) models capable of handling multiple tasks simultaneously. However, real-world applications often present data in a continuous stream of new domains and tasks, necessitating a lifelong learning approach.

Overview of the diverse domains and datasets used in the lifelong information extraction benchmark.

Problem Statement: Lifelong Information Extraction

Standard UIE models face two primary challenges in a lifelong learning scenario:

  1. Catastrophic Forgetting (CF): When a model is trained on a new task, it often loses the ability to perform previously learned tasks.
  2. Knowledge Transfer (KT): A robust lifelong learner should leverage prior knowledge to improve performance on subsequent tasks without requiring excessive memory for old data.

Methodology: Lottery Prompt Tuning (LPT)

To address these challenges, we propose Lottery Prompt Tuning (LPT). This method keeps a large Pre-trained Language Model (PLM) frozen and only optimizes a small set of "soft prompt" vectors. LPT draws inspiration from the Lottery Ticket Hypothesis, which posits that large neural networks contain sparse sub-networks capable of matching the performance of the full model.

Task-Aware Parameter Isolation

LPT applies this hypothesis to prompt tuning by learning a task-aware binary mask for each new task. This mask identifies a specific subset—a "lottery ticket"—of the shared prompt parameters. Once a parameter is assigned to a task, it is frozen for all future tasks, physically preventing the overwriting of old knowledge and eliminating catastrophic forgetting.

Illustration of the LPT mechanism, showing how task-specific prompt sub-networks (lottery tickets) are isolated and reused across sequential tasks.

Online Pruning and Selection

Instead of iterative pruning, LPT utilizes a "Straight-Through Estimator" to learn importance scores and masks concurrently during training. For novel or unseen tasks, the model employs a perplexity-based or gradient-based selection method to identify the most suitable mask from its library of previously learned tasks, facilitating zero-shot transfer.

Experimental Evaluation

The LPT framework was evaluated across a sequence of 13 IE datasets, covering four major extraction types: entity, relation, event, and aspect-based sentiment analysis. This benchmark includes diverse datasets such as ACE05, CoNLL04, and SemEval, testing the model's ability to scale across varied linguistic domains.

Key Results and Insights

LPT demonstrates significant improvements over existing lifelong learning baselines such as EWC, ER, and L2P:

  • Superior Performance: LPT achieved an Average F1 score of 76.914 across 13 tasks, a 3% improvement over the previous state-of-the-art (L2P).
  • Zero Forgetting: By design, LPT achieved a Backward Transfer (BWT) score of 0, indicating complete preservation of prior knowledge through parameter isolation.
  • Strong Generalization: In zero-shot settings, LPT significantly outperformed baselines (e.g., 32.69 F1 vs. 20.88 for L2P), validating the effectiveness of its mask-selection strategy for knowledge transfer.
  • Efficiency: The method is highly parameter-efficient, tuning only 0.097% of the model's parameters per task, making it suitable for deployment in resource-constrained environments.

Conclusion

Lottery Prompt Tuning (LPT) provides a scalable and efficient solution for lifelong information extraction. By isolating task-specific "lottery tickets" within a shared prompt space, LPT effectively balances the need for knowledge transfer with the requirement for zero forgetting. Future work will explore adaptive sparsity ratios and more complex prompt ensembling techniques to further enhance performance.

Reference: Liang, Z., et al. (2023). Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023).

Summary updated

BibTeX
@inproceedings{liang2023prompts,
  author = {Zujie Liang and Feng Wei and Yin Jie and Yuxi Qian and Zhenghong Hao and Bing Han},
  title = {{Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.16},
}
Figure