Abstract
International audience
Introduction
Few-Shot Event Detection (FSED) is a critical task in Natural Language Processing that involves identifying event "triggers"—words or phrases signifying specific events—using only a handful of labeled examples. Traditional approaches often struggle with the "null class" (non-events), leading to poor precision or recall. This paper introduces OUTFIT (OUT oF trIgger deTection), a novel framework that treats non-events as an out-of-domain detection problem and employs dynamic, sentence-specific similarity thresholding to achieve state-of-the-art results.
The Challenge of the "Null Class"
In standard prototypical networks, each event type is represented by a "prototype" vector. A major hurdle in FSED is the representation of the "null class" or non-event category. Standard models attempt to create a prototype for every word that is not an event. However, non-events are semantically heterogeneous—averaging the embeddings of words like "the," "apple," and "quickly" results in a blurry, incoherent representation. This makes it difficult for models to distinguish between legitimate event triggers and random background noise based on similarity alone.
The OUTFIT Approach
OUTFIT reimagines the task by defining what a non-event isn't, rather than what it is. The architecture abandons the traditional null prototype in favor of a more robust out-of-distribution (OOD) detection logic.
Key Technical Innovations
- No Null Prototype: The model only constructs prototypes for actual event types present in the support set.
- Contrastive Learning: Instead of standard cross-entropy, OUTFIT utilizes a contrastive loss function. This pushes trigger words closer to their respective prototypes while simultaneously pushing non-event words further away in the embedding space.
- Dynamic Thresholding via ECDF: Similarity score distributions vary significantly between sentences. A global threshold is often ineffective. OUTFIT employs the Empirical Cumulative Distribution Function (ECDF) to calculate a threshold dynamically for every sentence. It identifies an optimal probability threshold on a similar sentence in the support set and maps it back to the query sentence.
- Weighted BERT Representations: The system uses a pre-trained BERT-base encoder with a weighting strategy across layers to better capture contextual nuances.
Experimental Setup
The model was evaluated on three standard benchmarks: ACE 2005, MAVEN, and FewEvent. Experiments followed the "N-way, K-shot" protocol, specifically a 5-way, 5-shot setup where the model identifies 5 event types given 5 examples of each. To balance precision and recall, the authors incorporated Part-of-Speech (PoS) filtering, focusing on verbs, nouns, and adverbs.
Results and Analysis
OUTFIT significantly outperformed existing state-of-the-art models like PA-CRF and HCL-TAT, achieving an average 10-point improvement in micro F1-score across all datasets.
| Dataset | OUTFIT F1 | Previous Best F1 |
|---|---|---|
| ACE 2005 | 74.0 | 66.4 |
| MAVEN | 76.9 | 67.1 |
| FewEvent | 79.6 | 67.4 |
Ablation studies highlighted that the contrastive loss was essential for reducing variance, while the weighted BERT strategy provided the most substantial performance gain.
Conclusion
By treating non-events as an OOD problem and implementing dynamic thresholding, OUTFIT addresses the fundamental weaknesses of traditional prototypical networks in FSED. While a gap remains compared to theoretical "Oracle" performance (82-87 F1), OUTFIT sets a new standard for the field. Future work will explore more flexible filtering mechanisms and the application of this logic to Named Entity Recognition (NER) and Event Argument Extraction.
Summary updated