Abstract
Event factuality prediction (EFP) is the task of assessing the degree to which an event mentioned in a sentence has happened. For this task, both syntactic and semantic information are crucial to identify the important context words. The previous work for EFP has only combined these information in a simple way that cannot fully exploit their coordination. In this work, we introduce a novel graph-based neural network for EFP that can integrate the semantic and syntactic information more effectively. Our experiments demonstrate the advantage of the proposed model for EFP.
Introduction
In Natural Language Processing (NLP), Event Factuality Prediction (EFP) is the task of determining if an event described in text is a reality, a possibility, or a total fabrication. For example, in the sentence "She forgot to leave," the event "leave" did not actually happen, whereas in "She left," it certainly did. Current systems assign a score from -3 (certainly did not happen) to +3 (certainly happened).
The primary challenge lies in "cue words"—negations (not), modals (might, will), or belief verbs (thought, lied)—that can be far away from the "anchor word" (the event itself) in a sentence. Previous models processed syntactic structure (how words relate grammatically) and semantic structure (what words mean in context) separately, only combining them at the very end. This "late fusion" often fails to capture how these two types of information interact to reveal the true factuality of an event.
Methodology
The proposed model forces syntax and semantics to interact much earlier in the pipeline. The architecture consists of three main stages:
- Sentence Encoding: The model uses BERT to generate contextual word embeddings, which are then processed by a bidirectional LSTM to capture sequential flow.
- Structure Induction: Instead of relying solely on a standard dependency tree, the authors create a unified Affinity Matrix. This matrix integrates a Syntactic Matrix (derived from the sentence's dependency tree, as illustrated in Figure 1) and a Semantic Matrix (learned automatically from the LSTM hidden states). These are combined using a trade-off parameter ($\lambda$) to create a single graph representing both grammatical and meaning-based connections.
- Graph Convolutional Networks (GCN): The model runs GCN layers over this combined graph. This allows information from distant cue words to flow directly to the event anchor word through the most efficient path, whether that path is grammatical or semantic.
- Prediction: An attention mechanism focuses on the most relevant parts of the GCN-enriched representations to produce the final factuality score via regression.
Key Results
The proposed model was evaluated on four diverse benchmark datasets: FactBank, UW, Meantime, and UDS-IH2. As shown in Figure 2, the model outperformed previous state-of-the-art systems across almost all metrics:
- Pearson Correlation ($r$): The model showed significant improvements, particularly on the Meantime dataset, where it improved from a previous best of ~0.61 to 0.702.
- Mean Absolute Error (MAE): The model consistently achieved lower error rates. On FactBank, it achieved an MAE of 0.310, outperforming the previous best of 0.313.
- Multi-dataset Training: The model demonstrated a "transfer learning" effect; when trained on a union of multiple datasets, its performance improved further, suggesting the graph-based approach generalizes well across different annotation styles.
Conclusion
The research demonstrates that early interaction between syntax and semantics is key to accurate event factuality prediction. By merging these structures into a single graph, GCNs can filter out noise and focus on the most relevant linguistic cues.
One identified limitation is the reliance on external Dependency Parsing. Errors made by the parser can propagate through the model, potentially affecting performance in domains where parsers struggle (e.g., social media). Future work aims to develop methods that can automatically induce necessary sentence structures without requiring a pre-written dependency parser, increasing the model's flexibility across languages and domains.
Summary updated