NewsEdits 2.0: Learning the Intentions Behind Updating News

Alexander Spangher, Kung-Hsiang Huang, Hyundong Cho, Jonathan MayarXiv 2024 · DOI · arXiv

Introduction

In the rapidly evolving landscape of AI-driven information retrieval, the "shelf life" of a fact has become a critical concern. While news is often treated as a static record, it is in reality a fluid and evolving document. Researchers from USC and Salesforce AI have introduced NewsEdits 2.0, a framework designed to address the challenges posed by "factual fluidity" in news articles. By predicting which sentences are likely to be updated, this research aims to prevent Large Language Models (LLMs) from providing confident but outdated information.

The NewsEdits 2.0 Dataset and Taxonomy

At the core of this research is a new taxonomy of "edit intentions." Edits are categorized into four primary groups: Factual, Style, Narrative/Contextual, and Other. To build a high-quality training set, professional journalists were hired to annotate 9,200 sentence pairs, creating a gold-standard corpus.

Figure 1: An example of news article updates categorized by the NewsEdits 2.0 taxonomy, showing how initial warnings evolve into reports of impact.

Beyond the human-annotated set, the researchers developed a Silver-Labeled Corpus containing 4 million revision histories. This massive dataset was labeled using their best-performing model to facilitate large-scale training and robust evaluation.

Methodology: From Classification to Prediction

The research followed a multi-stage approach:

  1. Intent Modeling: A Longformer Encoder-Decoder (LED) model was trained to classify the reasons behind sentence changes. The model utilized not only the text but also discourse tags (e.g., "Main Event" vs. "Background") and argumentation labels.
  2. Predictive Modeling: The researchers shifted from classifying past edits to predicting future ones. The goal was to enable a model to analyze an initial version of an article and identify sentences likely to be revised.
  3. LLM Abstention: The team tested whether these update predictions could improve the reliability of LLMs like GPT-4. By providing "update likelihood" scores, they evaluated if the model would choose to abstain from answering questions based on potentially stale data.

Key Insights and Results

The study revealed several significant findings regarding how news evolves:

  • Linguistic Cues: Certain patterns serve as "red flags" for future updates. Sentences describing "Recent Events" or containing specific statistics are far more likely to change than descriptive or historical context.
  • Model Performance: The fine-tuned Longformer model achieved a Macro F1 score of 58.0 on the update prediction task, nearly matching the human benchmark of 58.6.
  • The Role of Context: Interestingly, the model performed almost as well when analyzing single sentences as it did with full article context, suggesting that the signal for factual fluidity is often embedded within the phrasing itself.
  • Feature Importance: Incorporating Discourse and Argumentation features provided a substantial boost, improving the F1 score for factual updates by over 17 points compared to a baseline model.

Practical Applications: LLM Reliability

One of the most impactful applications of this work is in Retrieval-Augmented Generation (RAG). Currently, LLMs lack an internal mechanism to detect if a retrieved document contains outdated facts.

Figure 4: Illustration of the challenge where an LLM may provide outdated information if it cannot recognize that its source data is in flux.

By integrating "Update Likelihood" scores, GPT-4 was able to reach near-oracle levels of accuracy in its decision to abstain. Instead of hallucinating based on stale information, the model correctly identified when it should say "I don't know."

Conclusion and Future Work

NewsEdits 2.0 demonstrates that reasoning about factual fluidity is not an emergent property of standard LLM pre-training but a specific skill that can be taught. While the current scope focuses on English and French news, future work may expand to other languages and explore non-factual edits, such as style and narrative shifts, to further support computational journalism.

Source Paper: NewsEdits 2.0: Learning the Intentions Behind Updating News (arXiv:2411.18811v1)

Summary updated

Abstract

As events progress, news articles often update with new information: if we are not cautious, we risk propagating outdated facts. In this work, we hypothesize that linguistic features indicate factual fluidity, and that we can predict which facts in a news article will update using solely the text of a news article (i.e. not external resources like search engines). We test this hypothesis, first, by isolating fact-updates in large news revisions corpora. News articles may update for many reasons (e.g. factual, stylistic, narrative). We introduce the NewsEdits 2.0 taxonomy, an edit-intentions schema that separates fact updates from stylistic and narrative updates in news writing. We annotate over 9,200 pairs of sentence revisions and train high-scoring ensemble models to apply this schema. Then, taking a large dataset of silver-labeled pairs, we show that we can predict when facts will update in older article drafts with high precision. Finally, to demonstrate the usefulness of these findings, we construct a language model question asking (LLM-QA) abstention task. We wish the LLM to abstain from answering questions when information is likely to become outdated. Using our predictions, we show, LLM absention reaches near oracle levels of accuracy.
BibTeX
@misc{spangher2024newsedits,
  author = {Alexander Spangher and Kung-Hsiang Huang and Hyundong Cho and Jonathan May},
  title = {{NewsEdits 2.0: Learning the Intentions Behind Updating News}},
  year = {2024},
  eprint = {2411.18811},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.18811},
}
Figure