Sources and Information Reliability Measures

Quentin ElsaesserIJCAI 2023 · DOI

TL;DR

This paper tackles the challenge of discerning truth from conflicting information sources online. The authors propose a novel method to simultaneously estimate the reliability of information sources and identify the most accurate facts, outperforming existing approaches on both real-world and synthetic datasets.

What Problem Does This Paper Address?

In today's digital age, we're inundated with information from various sources like social media and the web. This information is often contradictory, making it difficult to determine what's true. This paper addresses the critical problem of how to automatically assess the trustworthiness of these information sources and, in parallel, discover the actual facts amidst the noise. Existing "Truth Discovery" methods often focus on finding the truth but may not adequately quantify source reliability, or vice-versa. This work aims to bridge that gap by jointly optimizing both aspects.

Methods

The core of the proposed method is an iterative procedure that refines two key estimations: the reliability of each source and the truthfulness of each fact. The approach leverages a "voting-scoring rule" to assign scores to sources based on how their claimed facts fare against others for a given object. Two distinct normalization functions are used to ensure source reliabilities are between 0 and 1. The first normalization rewards sources that provide a lot of plausible information (facts that win votes), while the second focuses on the quality of information, rewarding sources that claim plausible facts even if they don't claim many. This iterative process continues until the source reliabilities stabilize. The authors also propose a set of properties that any such method should ideally satisfy, distinguishing between necessary and optional characteristics.

Datasets

The paper describes an experimental evaluation using both generated synthetic datasets and two real-world benchmarks: "Books" and "Flight." These real benchmarks are available at http://lunadong.com/fusionDataSets.htm.

Key Insights and Ideas

  • Joint Estimation is Key: The paper highlights the benefit of simultaneously estimating source reliability and fact truthfulness, suggesting these two problems are deeply intertwined.
  • Beyond Majority Vote: While majority voting is a starting point, the proposed scoring and normalization mechanisms offer a more nuanced way to reward sources and identify truth, especially in cases of ties or when the majority is wrong.
  • Properties for Robustness: Defining formal properties for truth discovery and reliability estimation provides a framework for understanding and comparing different methods, ensuring they behave in expected ways.

Key Results

The experimental evaluation demonstrated that the proposed methods outperform existing literature approaches (Truth Finder, Hubs and Authorities, Sums, Unbounded-Sums, and Voting) on both real and synthetic datasets for the Truth Discovery task. Notably, the methods are more effective at finding the truth when there are ties on an object or when the majority of sources provide incorrect information. Furthermore, the methods achieve good results in estimating source reliability, with the averaged difference between estimated and actual reliability being small. For one of the proposed methods, the estimated reliability was even identical to the probability of a source finding the truth when that probability exceeded 50%, and it also correctly identified the ranking of sources.

Limitations and Future Work

The authors acknowledge several avenues for future work. They aim to improve their methods by exploring alternative ways to update source reliability, such as gradual increases or decreases rather than immediate updates. Currently, the methods assume all objects are on the same topic; future work could extend this to handle multiple topics and dependencies between objects, allowing for the identification of domain-specific experts. The paper also notes that the current approach starts with no prior information about source reliability, and incorporating such prior knowledge could lead to different and potentially better results. Finally, they plan to adapt their methods to work with logical formulae instead of objects and to explore their application in new judgment aggregation methods.

Summary updated

BibTeX
@inproceedings{elsaesser2023sources,
  author = {Quentin Elsaesser},
  title = {{Sources and Information Reliability Measures}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/815},
}
Figure