Abstract
Document-level event factuality identification is a critical task in natural language processing that involves determining the degree to which a source commits to the truth of an event across an entire text. This paper introduces the DLEF corpus, a bilingual dataset for this task, and proposes a novel adversarial neural network model. The model utilizes dual-layered attention and adversarial training to effectively reconcile conflicting local cues and identify global document-level factuality. Experimental results on both English and Chinese datasets demonstrate that the proposed approach significantly outperforms traditional baselines.
Introduction
In Natural Language Processing (NLP), "event factuality" refers to the task of determining the degree of certainty a source expresses regarding an event's occurrence. For instance, in the sentence "The company might reach a deal," the event "reach" is categorized as a possibility (PS+), whereas in "The company denied it reached a deal," it is considered impossible (CT-).
While previous research has largely focused on sentence-level factuality, real-world documents often contain multiple, sometimes conflicting, mentions of the same event. A document might begin with speculation and conclude with a definitive denial. To accurately interpret such texts, systems must move beyond local cues to a global, document-level understanding. This paper addresses two primary challenges in this domain: the scarcity of annotated document-level factuality data and the need for models capable of integrating information across multiple sentences.
Methodology
The authors propose a specialized LSTM-based neural network architecture designed to capture an event's context throughout a document.
The model's core components include:
- Dual-Layered Attention:
- Intra-sequence attention: This mechanism focuses on finding specific "cue words" (e.g., denied, possibly) within a single sentence or dependency path that modify the event's factuality.
- Inter-sequence attention: When an event is mentioned multiple times, this layer determines which mention is the most "authoritative" for the final document-level verdict.
- Feature Fusion: The architecture integrates Syntactic Features (grammatical paths from cue words to the event) with Semantic Features (raw sentence text).
- Adversarial Training (AT): To enhance robustness against diverse writing styles, the authors introduce small, calculated perturbations to word embeddings during training. This regularizer forces the model to learn more generalized patterns rather than over-fitting to specific phrasing.
The DLEF Corpus
To facilitate research, the authors introduced the DLEF (Document-Level Event Factuality) corpus, the first dataset providing both sentence-level and document-level annotations.
- English: 1,727 documents sourced from China Daily and Sina Bilingual News.
- Chinese: 4,649 documents from Sina News.
- Factuality Labels: The schema includes CT+ (Certain/Positive), CT- (Certain/Negative), PS+ (Possible), PS- (Possible/Negative), and Uu (Underspecified).
Experimental Results
The proposed model, Att_2+AT, was evaluated against several baselines, including Maximum Entropy (MaxEnt) models and simple voting mechanisms.
As shown in the results table, the model achieved significant performance gains:
- English Performance: Reached a Macro-A F1 score of 76.28, a substantial improvement over the MaxEnt baseline of 56.40.
- Chinese Performance: Achieved a Macro-A F1 score of 81.64.
- Robustness: The model proved particularly effective at identifying CT- (impossible) events in Chinese, which are frequently used to debunk rumors in the news dataset.
Key Insights and Discussion
- Document Factuality vs. Sentence Majority: The study demonstrates that document-level factuality is not merely a majority vote of sentence-level labels. Often, a single authoritative denial can override multiple speculative mentions.
- Contextual Importance: Performance improved significantly when including sentences immediately surrounding an event mention, particularly in Chinese, indicating that discourse-level cues often cross sentence boundaries.
- Adversarial Training Benefits: The consistent improvement in F1 scores with AT confirms its utility in helping the model ignore noise and focus on salient features in complex news reports.
- Syntactic and Semantic Synergy: While dependency paths are useful for identifying direct modifiers, the best results were achieved by combining them with full semantic sentence representations.
Limitations and Future Work
The authors observed that joint optimization of sentence-level and document-level factuality did not yield improvements, likely due to the localized and sometimes contradictory nature of sentence-level cues. Additionally, the dataset exhibits a heavy skew toward CT+ events, reflecting the nature of news reporting. Future work will focus on modeling interactions between different events within a document and expanding the DLEF corpus.
Summary updated