Graph based Neural Networks for Event Factuality Prediction using Syntactic and Semantic Structures

Amir Pouran Ben Veyseh, Thien Huu Nguyen, Dejing Dou

Introduction

In natural language processing, Event Factuality Prediction (EFP) is the critical task of determining the degree to which a mentioned event actually occurred. This involves categorizing events as realities, possibilities, or fabrications. A significant challenge in EFP is the presence of "cue words" (e.g., might, didn't, forgot) that can modify or flip the factuality of an "anchor word" (the event itself), even when they are separated by many words in a sentence.

Previous approaches typically relied on either syntactic dependency trees or semantic sequences (via LSTMs). However, these models often employed "late fusion," combining these distinct information sources only at the final prediction stage. This paper argues that such a delay prevents the model from effectively filtering noise, proposing instead that syntax and semantics should interact much earlier in the neural architecture.

Methodology

The researchers developed a three-stage pipeline to facilitate early interaction between grammatical structure and contextual meaning:

  1. Sentence Encoding: Contextual word representations are generated using BERT embeddings, followed by a Bi-LSTM layer to capture the sequential flow of the text.
  2. Structure Induction: The core innovation is the creation of a hybrid graph. The model learns a Semantic Affinity Matrix ($A^{sem}$) based on LSTM hidden states and combines it with a Syntactic Matrix ($A^{syn}$) derived from a standard dependency tree. The final matrix is a weighted sum: $A = \lambda A^{sem} + (1-\lambda) A^{syn}$.
  3. Graph Convolutional Network (GCN): This hybrid matrix guides a GCN, allowing factuality information to flow across the most relevant edges, regardless of linear distance between words.
  4. Attention-based Prediction: An attention mechanism identifies the most relevant context words relative to the event anchor to produce a final factuality score on a scale of -3 to +3.

Experimental Setup

The model was evaluated on four major EFP benchmarks, each normalized to the [-3, +3] regression scale:

  • FactBank: A foundational dataset for event factuality.
  • UW: Focused on entity-event relations.
  • Meantime: A news-based corpus.
  • UDS-IH2: A large-scale dataset from the Universal Decompositional Semantics project.

Key Results

The proposed graph-based model achieved state-of-the-art (SOTA) performance across nearly all metrics, including Mean Absolute Error (MAE) and Pearson Correlation ($r$). Notably, on the Meantime dataset, the model reduced the MAE from a previous best of 0.340 to 0.204 when trained on multiple datasets.

Performance comparison of the proposed graph-based model against various baselines across four EFP benchmarks.

Key Insights and Analysis

The primary takeaway is that early interaction between syntax and semantics is superior to late-stage concatenation. Syntactic trees help the model locate distant modal verbs, while semantic structures help it ignore grammatical neighbors that are irrelevant to factuality.

Ablation studies confirmed that every component of the architecture contributes to its success. Removing either the syntactic or semantic component resulted in a measurable performance drop, illustrating their complementary nature.

Ablation study showing the impact of removing syntactic, semantic, and BERT components from the proposed model.

Conclusion and Future Work

While the model sets new benchmarks, it remains dependent on external dependency parsers. Errors in these parsers—common in informal or complex text—can propagate through the GCN. Future research will focus on methods to automatically induce optimal sentence structures for factuality prediction, potentially removing the need for pre-built parsers and increasing the model's robustness across different languages and domains.

Summary updated

Abstract

Event factuality prediction (EFP) is the task of assessing the degree to which an event mentioned in a sentence has happened. For this task, both syntactic and semantic information are crucial to identify the important context words. The previous work for EFP has only combined these information in a simple way that cannot fully exploit their coordination. In this work, we introduce a novel graph-based neural network for EFP that can integrate the semantic and syntactic information more effectively. Our experiments demonstrate the advantage of the proposed model for EFP.
BibTeX
@inproceedings{veyseh2019graph,
  author = {Amir Pouran Ben Veyseh and Thien Huu Nguyen and Dejing Dou},
  title = {{Graph based Neural Networks for Event Factuality Prediction using Syntactic and Semantic Structures}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1432},
}
Figure