Exploring Factual Entailment with NLI: A News Media Study

Guy Mor-Lan, Effi LeviNAACL 2024

TL;DR

This paper introduces "FactRel," a new annotation scheme for "factual entailment" that better captures the nuances of real-world news discourse than traditional Natural Language Inference (NLI). They found that factual relationships are far more common in news than strict semantic entailment, and that large language models like GPT-4 show promising few-shot learning capabilities for this task, highlighting the importance of world knowledge.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically verifying factual claims in news media. While Natural Language Inference (NLI) has been a popular tool for this, the authors argue that standard NLI, which focuses on semantic entailment (whether a hypothesis is necessarily true if a premise is true), is too restrictive for analyzing news discourse. News often involves more subtle factual relationships that don't fit neatly into strict entailment or contradiction. Furthermore, existing NLI datasets often use simpler, more generic sentences compared to the complex, information-rich sentences found in news articles. This gap means that NLI models might not effectively capture the factual relationships present in real-world news.

Methods

The core of the paper is the introduction of FactRel, a novel annotation scheme designed to model factual entailment rather than purely textual or semantic entailment. FactRel categorizes sentence pairs into three relations:

  • SUPPORT: Premise being true increases the plausibility of the hypothesis.
  • UNDERMINING: Premise being true decreases the plausibility of the hypothesis.
  • NEUTRAL: Premise has no effect on the hypothesis's plausibility.

They then created a dataset by annotating news article sentence pairs with both FactRel and traditional NLI labels to compare the two. They also experimented with various language models, including fine-tuning standard models and using GPT-4 in zero-shot and few-shot settings, to assess their performance on the FactRel task.

Datasets

  • Core Dataset: 1,507 sentence pairs sampled from 211 diverse English-language news articles (2020-2022). Pairs were sampled from the same article to increase the likelihood of non-neutral relationships.
  • MNLI Subset: 500 sentence pairs from the MNLI dataset, annotated with factual entailment to examine differences between the datasets.
  • Synthetic Dataset: Generated using GPT-4, creating 10 synthetic examples for each annotated sample in the core training set, resulting in 12,050 pairs.

Key Insights and Ideas

  • Factual vs. Semantic Entailment: The most striking finding is that factual relationships (SUPPORT/UNDERMINING) are significantly more prevalent in news media than strict semantic entailment or contradiction. This strongly suggests that NLI, as traditionally defined, is insufficient for analyzing news factuality.
  • World Knowledge is Crucial: Modeling factual relationships requires more than just understanding word meanings; it necessitates substantial background world knowledge and nuanced reasoning abilities. For example, understanding why one social network locking an account might support another doing so requires inferring common practices and policies.
  • GPT-4's Few-Shot Prowess: GPT-4 demonstrates remarkable performance in few-shot settings, achieving results comparable to medium-sized LMs fine-tuned on the entire dataset. This highlights its ability to leverage its broad world knowledge and reasoning capabilities with minimal task-specific examples.
  • Synthetic Data Benefits: Generating synthetic data with GPT-4 can improve the performance of some models, though the effect is not consistent across all configurations.

Key Results

  • In the core dataset, 93% of sentence pairs are NLI-neutral, while only 70% are factually neutral. This stark difference underscores the prevalence of factual relationships beyond semantic entailment.
  • 84% of factually supporting pairs and 63% of factually undermining pairs in the core dataset do not map to NLI entailment or contradiction. This further emphasizes the limitations of NLI for news analysis.
  • GPT-4, even in a zero-shot setting, performs competitively, and its 3-shot performance matches that of fine-tuned DeBERTa models.
  • Fine-tuning models on the core dataset yielded reasonable results, but the task remains challenging even for large pre-trained models.

Limitations and Future Work

  • Discrete Labels: The FactRel dataset uses discrete labels (SUPPORT, UNDERMINING, NEUTRAL). The authors acknowledge that a probabilistic framework could better quantify the amount of support or undermining, which is left for future research.
  • Model Performance: While GPT-4 shows promise, the task is still challenging, and performance improvements are desired.
  • Future Directions: The paper suggests exploring probabilistic models for factual relationships and further investigating the role of world knowledge and reasoning in LLMs for this task. The potential for synthetic data generation to augment training sets is also a promising avenue.

Summary updated

BibTeX
@inproceedings{morlan2024exploring,
  author = {Guy Mor-Lan and Effi Levi},
  title = {{Exploring Factual Entailment with NLI: A News Media Study}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure