BeLeaf: Belief Prediction as Tree Generation

John Murzaku, Owen RambowNAACL 2024 · DOI

TL;DR

This paper introduces BeLeaf, a novel approach to predicting event factuality by framing it as a linearized tree generation task. By representing complex nested source-and-target factuality relationships as trees, BeLeaf achieves state-of-the-art results on established benchmarks and offers an intuitive visualization tool.

What Problem Does This Paper Address?

Understanding the "factuality" or "belief" associated with an event is crucial for many downstream NLP tasks. Traditionally, event factuality prediction focused solely on the author's belief. However, texts often contain multiple sources (e.g., quoted experts, cited studies) that express their own beliefs about events, which may differ from the author's. Accurately capturing these nested beliefs is a significant challenge. Existing work either ignores these nested sources or uses representations that don't fully capture the hierarchical structure of these beliefs. This paper aims to address this by developing a method that can represent and predict factuality across multiple nested sources within a document.

Methods

The core innovation of BeLeaf lies in its linearized tree generation approach. Instead of treating factuality prediction as a classification or sequence labeling task, the authors reframe it as generating a structured tree representation of the source-and-target factuality.

  • Linearized Tree Representation: They convert the hierarchical structure of nested sources and their associated factuality beliefs into a linear sequence of tokens. This allows them to leverage powerful sequence-to-sequence models.
  • Encoder-Decoder Models: The system utilizes pre-trained transformer models, specifically Flan-T5 and GPT-3, as the backbone for this generation task. These models are fine-tuned to produce the linearized tree output.
  • Preprocessing: To handle document-level factuality, the system incorporates a coreference resolution module. This is crucial because it ensures that pronouns and other referring expressions are correctly linked to their antecedents, allowing for a more accurate representation of beliefs across sentences.
  • Postprocessing: A head-to-span module is used to extract the full syntactic span of events, rather than just the head word, providing richer context for the factuality prediction.
  • BeLeaf System: The paper also introduces a system called BeLeaf, which provides a user-friendly interface for inputting text and visualizing the generated factuality trees, including syntactic spans.

Datasets

The paper evaluates BeLeaf on two primary datasets:

  • FactBank: This corpus annotates event factuality at the sentence level, considering both the author's perspective and the perspectives of sources mentioned within the text.
  • Modal Dependency Corpus (MDP): This corpus represents factuality as a document-level modal dependency structure, with the author as the root and events and other sources as child nodes.

The authors also mention using their own split of FactBank from a previous work (Murzaku et al., 2022) and the MDP corpus split from Yao et al. (2021).

Key Insights and Ideas

  • Tree Generation as a Unified Framework: Framing factuality prediction as a linearized tree generation task proves to be a powerful and flexible approach, capable of handling complex nested structures.
  • Importance of Coreference Resolution: For document-level factuality, accurately resolving coreferences is essential for correctly attributing beliefs across sentences and sources.
  • Compression through Linearization: The linearized tree representation significantly compresses the token count compared to other representations, enabling more efficient end-to-end systems.
  • Interpretability via Visualization: The BeLeaf system's tree visualization offers a clear and intuitive way to understand the complex relationships between sources, events, and their associated factuality.

Key Results

  • State-of-the-Art Performance: BeLeaf achieves state-of-the-art results on both the FactBank and MDP corpora, outperforming previous methods.
  • FactBank Improvements: The Flan-T5-Tree model shows significant gains, particularly in the CT+ (certain) label, with a 9% absolute increase.
  • MDP Corpus Performance: The system beats the previous state-of-the-art on the MDP corpus by 3.7% on the development set and 3% on the test set.
  • GPT-3 vs. Flan-T5: While both models perform well, fine-tuned Flan-T5 generally outperforms GPT-3 on the MDP corpus, likely due to Flan-T5's instruction-tuning capabilities.

Limitations and Future Work

  • Limited to English: All experiments were conducted on English corpora. The authors plan to investigate multilingual source-and-target event factuality prediction in future work.
  • Potential Biases in LLMs: The current experiments do not account for potential biases present in large language models, which could affect factuality predictions, especially concerning specific sources or groups. This is an area for future investigation.
  • Single Run for GPT-3: Due to cost constraints, GPT-3 experiments were performed with a single run, meaning averages and standard deviations are not reported.
  • Future Work: The authors express interest in exploring multilingual generalization and investigating biases in LLMs for factuality prediction.

Summary updated

BibTeX
@inproceedings{murzaku2024beleaf,
  author = {John Murzaku and Owen Rambow},
  title = {{BeLeaf: Belief Prediction as Tree Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-demo.10},
}
Figure