Introduction
In natural language processing, Event Factuality Prediction (EFP) is the critical task of determining the degree to which a mentioned event actually occurred. This involves categorizing events as realities, possibilities, or fabrications. A significant challenge in EFP is the presence of "cue words" (e.g., might, didn't, forgot) that can modify or flip the factuality of an "anchor word" (the event itself), even when they are separated by many words in a sentence.
Previous approaches typically relied on either syntactic dependency trees or semantic sequences (via LSTMs). However, these models often employed "late fusion," combining these distinct information sources only at the final prediction stage. This paper argues that such a delay prevents the model from effectively filtering noise, proposing instead that syntax and semantics should interact much earlier in the neural architecture.
Methodology
The researchers developed a three-stage pipeline to facilitate early interaction between grammatical structure and contextual meaning:
- Sentence Encoding: Contextual word representations are generated using BERT embeddings, followed by a Bi-LSTM layer to capture the sequential flow of the text.
- Structure Induction: The core innovation is the creation of a hybrid graph. The model learns a Semantic Affinity Matrix ($A^{sem}$) based on LSTM hidden states and combines it with a Syntactic Matrix ($A^{syn}$) derived from a standard dependency tree. The final matrix is a weighted sum: $A = \lambda A^{sem} + (1-\lambda) A^{syn}$.
- Graph Convolutional Network (GCN): This hybrid matrix guides a GCN, allowing factuality information to flow across the most relevant edges, regardless of linear distance between words.
- Attention-based Prediction: An attention mechanism identifies the most relevant context words relative to the event anchor to produce a final factuality score on a scale of -3 to +3.
Experimental Setup
The model was evaluated on four major EFP benchmarks, each normalized to the [-3, +3] regression scale:
- FactBank: A foundational dataset for event factuality.
- UW: Focused on entity-event relations.
- Meantime: A news-based corpus.
- UDS-IH2: A large-scale dataset from the Universal Decompositional Semantics project.
Key Results
The proposed graph-based model achieved state-of-the-art (SOTA) performance across nearly all metrics, including Mean Absolute Error (MAE) and Pearson Correlation ($r$). Notably, on the Meantime dataset, the model reduced the MAE from a previous best of 0.340 to 0.204 when trained on multiple datasets.
Key Insights and Analysis
The primary takeaway is that early interaction between syntax and semantics is superior to late-stage concatenation. Syntactic trees help the model locate distant modal verbs, while semantic structures help it ignore grammatical neighbors that are irrelevant to factuality.
Ablation studies confirmed that every component of the architecture contributes to its success. Removing either the syntactic or semantic component resulted in a measurable performance drop, illustrating their complementary nature.
Conclusion and Future Work
While the model sets new benchmarks, it remains dependent on external dependency parsers. Errors in these parsers—common in informal or complex text—can propagate through the GCN. Future research will focus on methods to automatically induce optimal sentence structures for factuality prediction, potentially removing the need for pre-built parsers and increasing the model's robustness across different languages and domains.
Summary updated