Does It Happen? Multi-hop Path Structures for Event Factuality Prediction with Graph Transformer Networks

Duong Le, Thien Huu NguyenW-nut 2021 · DOI

TL;DR

This paper introduces a novel deep learning model for Event Factuality Prediction (EFP) that leverages multi-hop paths and heterogeneous edge types within sentence structures. By incorporating Graph Transformer Networks (GTNs) and an Information Bottleneck approach, the model significantly improves the accuracy of determining how likely an event mentioned in text has actually occurred.

What Problem Does This Paper Address?

Event Factuality Prediction (EFP) aims to determine the factual degree of an event mention in text, essentially answering "did this event really happen?" This is crucial for applications like knowledge base construction, where distinguishing between factual and non-factual events is essential. Current deep learning models for EFP often rely on syntactic and semantic structures of sentences to identify important "cue words" that signal factuality. However, a major limitation is that these models primarily focus on one-hop connections (direct relationships) between words, neglecting the richer, indirect relationships that can exist across multiple words. For instance, understanding the factuality of "involvement" might require connecting it to "family members" and then to "brother," a multi-hop path that direct connections miss. This paper addresses this gap by proposing a model that explicitly captures these multi-hop paths and heterogeneous (syntactic and semantic) connections.

Methods

The proposed model introduces several key innovations:

  • Customized Syntactic and Semantic Structures: Unlike previous methods that used generic syntactic structures, this work customizes them based on the event trigger word. This involves calculating the shortest path distance between words and the trigger in the dependency tree to inform the importance scores. Similarly, semantic structures are also conditioned on the trigger word to filter relevant information.
  • Graph Transformer Networks (GTNs) for Multi-hop Reasoning: The core of the model uses GTNs to combine the initial syntactic and semantic structures. GTNs are adept at learning multi-hop paths by iteratively multiplying intermediate structures, allowing them to capture complex relationships between words that go beyond direct connections. This also enables the modeling of heterogeneous edge types (combining syntactic and semantic information).
  • Information Bottleneck (IB) for Regularization: To combat overfitting, especially with limited EFP datasets, the authors incorporate an Information Bottleneck approach. This encourages the model to retain only the most relevant information for factuality prediction while minimizing mutual information with the original input representations, thus improving generalization.
  • BERT Embeddings: The model leverages pre-trained BERT embeddings as the initial input vectors for each word, providing rich contextual representations.

Datasets

The paper evaluates the proposed model on four standard EFP datasets:

  • FactBank
  • UW
  • Meantime
  • UDS-IH2

The authors note that they use preprocessed versions of these datasets, including dependency trees, from previous work to ensure a fair comparison.

Key Insights and Ideas

  • Multi-hop paths are crucial for EFP: The paper strongly argues that indirect, multi-hop relationships between words are as important as direct ones for accurately predicting event factuality.
  • Trigger-word conditioning enhances structures: Customizing syntactic and semantic structures based on the specific event trigger word significantly improves their relevance and effectiveness.
  • GTNs excel at capturing complex sentence structures: The use of GTNs allows for the modeling of intricate, multi-hop reasoning paths and heterogeneous edge types, which are vital for understanding nuanced factuality.
  • Information Bottleneck improves generalization: Regularizing the model with an IB approach effectively prevents overfitting and leads to better performance on unseen data.

Key Results

The proposed model, "SynSem-Customization+MultiHop+GCN+IB+BERT," achieves state-of-the-art performance on the evaluated datasets. Specifically:

  • It significantly outperforms baseline models, including previous state-of-the-art methods, across different performance measures (MAE and Pearson Correlation).
  • When trained on multiple datasets, the model shows particularly strong results, demonstrating its ability to generalize well.
  • Ablation studies confirm the importance of each component: customized structures, GTN for multi-hop reasoning, and the Information Bottleneck regularization all contribute to the overall performance gains.

Limitations and Future Work

  • Unknown Words: The authors identify "unknown words" (words not present in the training data) as a significant source of errors, even with BERT embeddings. They hypothesize that models lack sufficient training signals to properly encode these words.
  • Future Directions: The paper suggests focusing on improving the handling of unknown words for EFP. Additionally, the authors plan to extend their proposed model to related tasks within EFP.

Summary updated

BibTeX
@inproceedings{le2021does,
  author = {Duong Le and Thien Huu Nguyen},
  title = {{Does It Happen? Multi-hop Path Structures for Event Factuality Prediction with Graph Transformer Networks}},
  booktitle = {W-nut},
  year = {2021},
  doi = {10.18653/v1/2021.wnut-1.6},
}
Figure