Cost-sensitive Regularization for Label Confusion-aware Event Detection

Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun

Introduction

Event Detection (ED) is a fundamental task in Information Extraction, requiring systems to identify "trigger" words that signify specific events. For instance, in the sentence "The company was acquired," an ED system should recognize "acquired" as a trigger for a Business:Merge-Org event. Despite significant progress with neural networks, a persistent challenge remains: label confusion. This occurs when models struggle to distinguish between semantically similar event types or fail to differentiate an event trigger from a non-event (NIL) word.

The Problem: Label Confusion

The authors identify two primary forms of label confusion that hinder performance in supervised ED models:

  1. Trigger/NIL Confusion: This involves mistaking a common word for an event trigger or vice versa. For example, the word "give" might trigger a transaction event in one context but have no event-related meaning (NIL) in another.
  2. Sibling Sub-type Confusion: This occurs when a model confuses closely related event types, such as mistaking a Contact:Phone-Write event for a Contact:Meet event.

Standard training objectives, like cross-entropy loss, treat all misclassifications as equally detrimental. They focus on maximizing the likelihood of the correct label without explicitly discouraging the model from falling into these specific semantic "confusion traps."

Confusion matrix illustrating the label confusion problem in event detection, where many event types are frequently misclassified as NIL or other similar types.

Methodology: Cost-sensitive Regularization

To address this, the paper proposes Cost-sensitive Regularization ($L_{CS}$). This method augments the standard cross-entropy loss with a weighted penalty term that specifically targets confusing label pairs. The total loss function is defined as:

$$\mathcal{L}(\theta) = \mathcal{L}_{CE}(\theta) + \lambda\mathcal{L}_{CS}(\theta)$$

The core innovation is the estimation of the "cost" of mislabeling. The authors introduce two distinct estimators:

  • Population-level Estimator (CR-POP): This estimator calculates costs based on global confusion statistics derived from a development set. If the model historically confuses Label A for Label B, the penalty for that specific pair is increased during training.
  • Instance-level Estimator (CR-INS): This dynamic approach uses the model’s own current predicted probability for an incorrect class as a proxy for confusion risk. If the model shows a high "temptation" to pick a wrong label for a specific instance, the penalty for that instance is increased in real-time.

Experimental Setup and Results

The proposed method was evaluated on the TAC-KBP 2017 Event Nugget Detection datasets for both English and Chinese. The authors compared their regularization technique against standard baselines and common techniques for handling imbalanced data, such as Focal Loss and Under-sampling.

Performance comparison of different loss functions and regularization methods on English and Chinese datasets using LSTM and DMCNN architectures.

The results, summarized in the table above, show that the cost-sensitive approach—particularly the instance-level estimator (CR-INS)—consistently achieves state-of-the-art (SOTA) performance. On the English dataset using an LSTM backbone, the F1 score improved significantly from 46.70 (standard cross-entropy) to 53.71 (CR-INS).

Key Insights

  • Semantics over Statistics: The study suggests that label confusion is driven more by semantic similarity than by simple data imbalance. Even when compared to methods specifically designed for imbalance (like Focal Loss), the cost-sensitive approach yielded superior results.
  • Dynamic Learning: CR-INS generally outperformed CR-POP. By updating the cost matrix every batch rather than every epoch, CR-INS provides a more granular and accurate target for the model to optimize against.
  • Model Portability: The regularization term is model-agnostic. The authors demonstrated its effectiveness by applying it to both Convolutional Neural Networks (DMCNN) and Recurrent Neural Networks (LSTM), observing significant gains across both architectures and languages.

Conclusion

By explicitly modeling the costs associated with label confusion, the proposed regularization technique provides a robust solution to one of the most persistent challenges in Event Detection. The method not only achieves state-of-the-art results on competitive benchmarks but also offers a portable framework that can be integrated into various neural architectures. Future work may explore applying this cost-sensitive loss to other NLP tasks characterized by high semantic similarity between labels, such as fine-grained entity typing.

Summary updated

Abstract

In supervised event detection, most of the mislabeling occurs between a small number of confusing type pairs, including trigger-NIL pairs and sibling sub-types of the same coarse type. To address this label confusion problem, this paper proposes cost-sensitive regularization, which can force the training procedure to concentrate more on optimizing confusing type pairs. Specifically, we introduce a cost-weighted term into the training loss, which penalizes more on mislabeling between confusing label pairs. Furthermore, we also propose two estimators which can effectively measure such label confusion based on instance-level or population-level statistics. Experiments on TAC-KBP 2017 datasets demonstrate that the proposed method can significantly improve the performances of different models in both English and Chinese event detection.
BibTeX
@inproceedings{lin2019cost,
  author = {Hongyu Lin and Yaojie Lu and Xianpei Han and Le Sun},
  title = {{Cost-sensitive Regularization for Label Confusion-aware Event Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1521},
}
Figure