Towards Generative Event Factuality Prediction

John Murzaku, Tyler Osborne, Amittai Aviram, Owen RambowACL 2023 · DOI

Introduction

Event Factuality Prediction (EFP) is a critical task in Natural Language Processing that involves determining the degree to which an author or a mentioned source commits to the truth of an event. Unlike objective fact-checking, EFP focuses on linguistic framing: does the writer present an event as a certainty, a possibility, or a negation?

While traditional systems have primarily focused on the author's perspective, real-world communication often involves complex nested perspectives (e.g., "John said that Mary believes..."). This paper introduces a novel generative approach to EFP, framing it as a text-to-text task using the Flan-T5 model to capture these multi-layered attributions.

The Challenge of Nested Perspectives

Capturing factuality across different layers of attribution is historically difficult. Consider the sentence: "Michael Wu said he’d double his money." In this context, the author presents the act of "saying" as a fact, but the act of "doubling" is presented as unknown by the author, while Michael Wu himself presents it as a fact.

Example of factuality triplets for nested sources, showing source, target event, and the resulting factuality value.

Previous systems often relied on pre-identified "gold" event heads, which limits their utility in end-to-end applications. This research addresses this by treating EFP as a structured generation task, where the model outputs triplets consisting of the source, the target event, and the factuality value.

Methodology: A Generative Approach

The researchers utilized Flan-T5, an encoder-decoder model, to generate structured text representing factuality. Several technical innovations were key to the system's success:

  • Attribute-Value (AV) Formatting: The team found that explicitly labeling keys (e.g., source = Wu, target = double, factuality = true) significantly outperformed simple tuple formats. This structure provides the model with clearer contextual cues during decoding.
  • Multi-task Learning (MTL): By training the model simultaneously on diverse datasets—including CommitmentBank (CB), UW, and MPQA—the model learned to better navigate the nuances of linguistic commitment.
  • Output Normalization: To mitigate the risk of generative "hallucinations" (such as minor spelling variations), the researchers applied Levenshtein distance-based normalization to ensure generated tokens matched the original text.

Key Results and Insights

The generative framework established new state-of-the-art (SOTA) benchmarks for end-to-end EFP. On the FactBank corpus, the model achieved a 0.694 Macro-F1 score for author-only perspectives. More impressively, it established a robust baseline for the more difficult source-and-target task.

Performance comparison of the FBST baseline against various Multi-task Learning (MTL) combinations, showing significant gains in Triplet F1.

As shown in the experimental results, the inclusion of auxiliary datasets through multi-task learning (particularly the combination of FBST and FBAO) pushed the Triplet F1 score from a baseline of 0.535 to 0.684*. The model proved particularly adept at identifying "Certain Factual" (CT+) and "Unknown" (UU) labels, though nested source prediction remains a harder challenge than author-only prediction, with a observed 13.3% performance gap.

Conclusion and Future Directions

By moving from classification to generation, this research demonstrates that the structure of model outputs is as vital as the architecture itself. While the current model excels at identifying the last source in a chain, future work will look toward capturing full attribution chains (e.g., A says that B says that C happened). Additionally, as models scale, addressing inherent biases in how LLMs perceive different sources remains a critical ethical priority.

Summary updated

BibTeX
@inproceedings{murzaku2023towards,
  author = {John Murzaku and Tyler Osborne and Amittai Aviram and Owen Rambow},
  title = {{Towards Generative Event Factuality Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.44},
}
Figure