Abstract
Information retrieval may suggest a document, and information extraction may tell us what it says, but which information sources do we trust and which assertions do we believe when different authors make conflicting claims? Trust algorithms known as fact-finders attempt to answer these questions, but consider only which source makes which claim, ignoring a wealth of background knowledge and contextual detail such as the uncertainty in the information extraction of claims from documents, attributes of the sources, the degree of similarity among claims, and the degree of certainty expressed by the sources. We introduce a new, generalized fact-finding framework able to incorporate this additional information into the fact-finding process. Experiments using several state-of-the-art fact-finding algorithms demonstrate that generalized fact-finders achieve significantly better performance than their original variants on both semi-synthetic and real-world problems.
Introduction
In the modern information landscape, we are inundated with data from sources of varying reliability. When sources conflict—for instance, when two websites provide different birthplaces for a public figure—deciding which to believe becomes a critical challenge. Existing "fact-finder" algorithms address this by iteratively calculating source trustworthiness and claim belief. However, these models typically operate on a simple bipartite graph, considering only the raw links between sources and claims. They often ignore vital contextual metadata, such as extraction uncertainty, claim similarity, and source attributes (e.g., reputation or group membership).
The Generalized Framework
The core contribution of this work is the transition from a simple bipartite graph (Sources $\to$ Claims) to a weighted k-partite graph. This allows for the integration of diverse background knowledge into the fact-finding process.
The framework introduces several key innovations:
- Weighted Assertions: Instead of binary assertions, the framework uses a weight $\omega(s, c) \in [0, 1]$, combining extraction uncertainty, source-expressed certainty, similarity-based support, and group-based support.
- K-Partite Layers: Additional layers represent groups and attributes directly. For example, a "Group" layer can sit above the "Source" layer to incorporate priors like administrative status.
- Algorithm Rewriting: Systematic rules are provided to adapt existing linear and log-linear algorithms (e.g., Sums, TruthFinder, and Investment) to these new weights.
- Smoothing ($D^{smooth}$): A smoothing function prevents individual sources from being unfairly penalized by mediocre group averages, scaling group impact based on individual evidence.
Experimental Setup
The framework was evaluated on two primary real-world datasets:
- Population Dataset: 44,761 claims regarding city populations from Wikipedia infoboxes, involving over 171,000 editors. Ground truth was established via official census data.
- Books Dataset: 14,287 claims about book authorship from 894 bookstore websites, verified against physical book covers.
Results and Discussion
The generalized framework consistently outperformed standard fact-finding algorithms across both datasets. By treating claims as related rather than strictly mutually exclusive (e.g., recognizing that "1950" is closer to "1951" than to "2010"), the algorithms preserved more source trustworthiness even when claims were slightly off.
Key results include:
- Accuracy Boost: On the Wikipedia Population dataset, incorporating assertion certainty allowed the "Investment" algorithm to reach 90.00% accuracy, surpassing previous state-of-the-art methods.
- Robustness: Generalized fact-finders showed significant resilience to noise, outperforming standard variants by 5-10 percentage points in scenarios with high extraction error rates.
- Group Layer Efficacy: Adding group layers (e.g., distinguishing Wikipedia Administrators) improved performance for nearly every algorithm, particularly when combined with the $D^{smooth}$ function.
Conclusion
Generalized fact-finding provides a powerful mechanism for incorporating contextual metadata into trust decisions. While it introduces some computational complexity and requires metadata availability, the significant gains in accuracy and robustness make it a compelling approach for real-world information integration. Future work may explore automated discovery of groups and similarity functions to further reduce dependency on pre-defined metadata.
Summary updated