Word-level Commonsense Knowledge Selection for Event Detection

Shuai Yang, Yu Hong, Shiming He, Qingting Xu, Jianmin YaoCOLING 2024

TL;DR

This paper introduces a novel approach to Event Detection (ED) by leveraging commonsense knowledge to improve word sense disambiguation. They propose a "Context-specific Knowledge Selector" (CKS) that intelligently picks relevant commonsense facts from ConceptNet, enhancing the understanding of trigger words and leading to significant performance gains on the ACE-2005 benchmark.

What Problem Does This Paper Address?

Event Detection (ED) is a crucial task in Natural Language Processing (NLP) that involves identifying specific words (triggers) in text that signal an event and classifying the type of event they represent. A major challenge in ED is accurately understanding the meaning of these trigger words, especially when they have multiple senses (word sense disambiguation). Traditional ED models often struggle with this, leading to errors. This paper addresses the problem of "knowledge deficiency" in ED by proposing a method to inject relevant commonsense knowledge into the model, thereby improving its ability to disambiguate word senses and correctly identify event types.

Methods

The core of this paper's contribution lies in its Context-specific Knowledge Selector (CKS). Here's a breakdown of their approach:

  • Backbone: They use a standard BERTlarge model as the encoder to generate contextualized word embeddings.
  • Knowledge Source: They utilize ConceptNet, a large knowledge graph containing commonsense knowledge, to retrieve relevant facts about words.
  • Knowledge Retrieval: For each content word (nouns, verbs, adjectives, adverbs), they retrieve triples from ConceptNet (e.g., {war, Causes, death}).
  • Knowledge Formation: These triples are converted into natural language sentences (denotations), like "War causes death."
  • Context-specific Knowledge Selector (CKS): This is the novel component. Instead of using all retrieved commonsense facts, CKS is trained to select only the most relevant ones for a given word in its specific context. CKS is built using a BERTbase encoder and a classifier. It's trained on a multi-choice question-answering task where it must pick the correct commonsense denotation from a list.
  • Knowledge Integration: The selected commonsense knowledge representation is then concatenated with the word's contextualized embedding from BERTlarge and fed into the discriminator for event classification.
  • ChatGPT for Training Data: To create training data for CKS, they leverage ChatGPT to predict the relevant commonsense knowledge for a subset of their data, acknowledging its power but also its cost-prohibitive nature for large-scale training.

Datasets

  • ACE-2005: This is the primary dataset used for training and evaluating their Event Detection model. It's a widely used benchmark in the field.
  • ConceptNet: Used as the source of commonsense knowledge.
  • ChatGPT-generated data: A small portion of the ACE-2005 dataset was used to generate pseudo-ground truth for training the CKS model, highlighting a practical approach to leveraging powerful LLMs for data creation.

Key Insights and Ideas

  • Commonsense is Key for Word Sense: The paper strongly advocates that commonsense knowledge is not just helpful but crucial for accurately disambiguating word senses in the context of Event Detection.
  • Context Matters for Commonsense: Not all commonsense knowledge is useful for every context. The ability to select contextually relevant commonsense facts is more effective than using a broad set of facts.
  • Intelligent Selection Outperforms Randomness: Their CKS model significantly outperforms a baseline that randomly selects commonsense knowledge, demonstrating the importance of their selection mechanism.
  • ChatGPT as a Powerful, but Costly, Tool: The paper shows ChatGPT can generate high-quality training data for commonsense selection, but they propose an offline CKS as a more scalable and cost-effective alternative for large-scale training.

Key Results

  • The proposed model, BERTlarge + CKS, achieved an F1-score of 78.3% on the ACE-2005 dataset.
  • This represents a 2% improvement over the BERTlarge baseline (76.3% F1).
  • The improvement was driven by a higher precision (3.8% growth rate), indicating the model made fewer incorrect predictions.
  • Their approach outperformed other classification-based ED models like APEX.
  • While comparable to some knowledge enrichment methods (like EKD), their model is noted as being more "vest-pocket like" due to its simpler training process for the knowledge enrichment part.

Limitations and Future Work

  • Real-time Knowledge Retrieval: The current model requires real-time retrieval of commonsense knowledge during testing, which could be a bottleneck for very large-scale or low-latency applications.
  • Out-of-Vocabulary (OOV) Problem: The paper acknowledges the out-of-vocabulary problem for trigger words, which is not fully addressed by their current approach.
  • Future Work: The authors plan to connect their model with a commonsense knowledge generator. This could potentially address the OOV problem and lead to a more generalized ED model.

Summary updated

BibTeX
@inproceedings{yang2024word,
  author = {Shuai Yang and Yu Hong and Shiming He and Qingting Xu and Jianmin Yao},
  title = {{Word-level Commonsense Knowledge Selection for Event Detection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure