Investigating Rumor News Using Agreement-Aware Search

Jingbo Shang, Tianhang Sun, Jiaming Shen, Xingbang Liu, Anja Gruenheid, Flip Korn, Ádám D. Lelkes, Cong Yu, Jiawei Han

Abstract: Recent years have witnessed a widespread increase of rumor news generated by humans and machines. Therefore, tools for investigating rumor news have become an urgent necessity. One useful function of such tools is to see ways a specific topic or event is represented by presenting different points of view from multiple sources. In this paper, we propose Maester, a novel agreement-aware search framework for investigating rumor news. Given an investigative question, Maester will retrieve related articles to that question, assign and display top articles from agree, disagree, and discuss categories to users. Splitting the results into these three categories provides the user a holistic view towards the investigative question. We build Maester based on the following two key observations: (1) relatedness can commonly be determined by keywords and entities occurring in both questions and articles, and (2) the level of agreement between the investigative question and the related news article can often be decided by a few key sentences. Accordingly, we use gradient boosting tree models with keyword/entity matching features for relatedness detection, and leverage recurrent neural network to infer the level of agreement. Our experiments on the Fake News Challenge (FNC) dataset demonstrate up to an order of magnitude improvement of Maester over the original FNC winning solution, for agreement-aware search.

Introduction

The proliferation of rumor news on the internet has created a significant challenge for users seeking factual clarity. Traditional search engines and Question Answering (QA) systems often fall short in this context. While QA systems are optimized for single-answer fact-finding, standard search engines return flat lists of results that obscure the diversity of perspectives on controversial topics.

To bridge this gap, we introduce Maester, an "agreement-aware" search framework. Maester moves beyond simple ranking by categorizing relevant news articles into three distinct stances: agree, disagree, and discuss. This multi-perspective approach allows journalists, fact-checkers, and general users to quickly grasp the landscape of a rumor and identify conflicting viewpoints.

The Maester Framework

Maester is designed as a two-step pipeline, grounded in the observation that identifying what an article is about requires different features than identifying how it stands on a topic.

The Maester system architecture, showing the two-step pipeline for relatedness and agreement classification.

Step 1: Relatedness Classification

The first stage filters the document corpus to find articles related to the investigative query. Using a gradient boosting tree model (XGBoost), Maester analyzes keyword and named-entity overlaps. By focusing on these high-level features, the system can efficiently distinguish between articles discussing the same entities (e.g., "Robert Plant" and "Led Zeppelin") and those that are entirely unrelated.

Step 2: Agreement Detection

Once an article is deemed related, Maester must determine its stance. This is a more complex task because both agreeing and disagreeing articles share the same entities. Stance is often buried in subtle linguistic cues—adjectives, negations, and specific phrasing—typically found in the "inverted pyramid" structure of news writing.

Detailed architecture of the Agreement Classifier, utilizing an RNN with attention to compare key article sentences against the query.

Maester employs an attention-based Recurrent Neural Network (RNN) for this task. To optimize performance and reduce noise, the system uses a "top-3 sentence trick": it identifies the three sentences in the article most semantically similar to the query and feeds only those into the RNN. This allows the model to focus on the most relevant evidence for stance detection.

Experimental Results

We evaluated Maester using the Fake News Challenge (FNC-1) dataset, which contains over 75,000 labeled headline-article pairs. The dataset is highly imbalanced, with the disagree class representing less than 3% of the data, providing a rigorous test for identifying minority viewpoints.

Performance Gains

Maester significantly outperformed the FNC-1 winning solution across several metrics:

  • Ranking Quality: Overall ranking quality (Average NDCG) improved by 20%.
  • Surfacing Dissent: For the rare and difficult disagree class, Maester achieved a 7-fold improvement in ranking performance (19.13% vs 2.58% NDCG@3).
  • Efficiency: The system is highly optimized for real-time use, processing a query-article pair in approximately 5.86 milliseconds and presenting a full UI in roughly 0.16 seconds.

Conclusion

Maester demonstrates that an agreement-aware approach to search can provide much-needed clarity in the era of rumor news. By decoupling relevance from stance and leveraging attention-based RNNs on key sentences, Maester provides a scalable and accurate tool for multi-perspective news investigation. Future work will focus on expanding stance categories and automating the discovery of controversial topics for investigation.

Summary updated

Abstract

Recent years have witnessed a widespread increase of rumor news generated by humans and machines. Therefore, tools for investigating rumor news have become an urgent necessity. One useful function of such tools is to see ways a specific topic or event is represented by presenting different points of view from multiple sources. In this paper, we propose Maester, a novel agreement-aware search framework for investigating rumor news. Given an investigative question, Maester will retrieve related articles to that question, assign and display top articles from agree, disagree, and discuss categories to users. Splitting the results into these three categories provides the user a holistic view towards the investigative question. We build Maester based on the following two key observations: (1) relatedness can commonly be determined by keywords and entities occurring in both questions and articles, and (2) the level of agreement between the investigative question and the related news article can often be decided by a few key sentences. Accordingly, we use gradient boosting tree models with keyword/entity matching features for relatedness detection, and leverage recurrent neural network to infer the level of agreement. Our experiments on the Fake News Challenge (FNC) dataset demonstrate up to an order of magnitude improvement of Maester over the original FNC winning solution, for agreement-aware search.
BibTeX
@inproceedings{shang2018investigating,
  author = {Jingbo Shang and Tianhang Sun and Jiaming Shen and Xingbang Liu and Anja Gruenheid and Flip Korn and Ádám D. Lelkes and Cong Yu and Jiawei Han},
  title = {{Investigating Rumor News Using Agreement-Aware Search}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2018},
  url = {https://openreview.net/forum?id=ZxEJKcC-kVf},
}
Figure