Abstract
The proliferation of large language models (LLMs) has introduced unprecedented challenges in fake news detection due to benchmark data contamination (BDC), where evaluation benchmarks are inadvertently memorized during pre-training, leading to inflated performance metrics. This paper introduces TripleFact, a novel evaluation framework designed to mitigate BDC risk while prioritizing real-world applicability. TripleFact integrates three components: Human-Adversarial Preference Testing (HAPT), Real-Time Web Agent with Asynchronous Validation (RTW-AV), and Entity-Controlled Virtual Environment (ECVE). Through experiments on 17 state-of-the-art LLMs, TripleFact demonstrates superior contamination resistance. Results reveal that BDC artificially inflates performance by up to 23% in conventional evaluations, while the TripleFact Score (TFS) remains stable within 4% absolute error under controlled contamination.
The Challenge of Benchmark Data Contamination
As Large Language Models (LLMs) are increasingly deployed for information verification, the integrity of their evaluation becomes paramount. Traditional fake news detection benchmarks rely on static, historical datasets. However, because modern LLMs are pre-trained on massive, open-domain web corpora, they often encounter these evaluation datasets during their training phase. This phenomenon, known as Benchmark Data Contamination (BDC), allows models to "cheat" by recalling memorized labels rather than applying genuine reasoning.
Furthermore, fake news is inherently temporal. A model trained on data up to late 2023 cannot reliably detect misinformation emerging in 2025 without external tools. Current benchmarks also fail to account for entity bias—the tendency of models to associate specific names or organizations with "fake" or "real" labels based on training frequency rather than factual content.
The TripleFact Framework
To address these limitations, TripleFact shifts evaluation from static datasets to a dynamic, adversarial, and entity-neutral environment.
TripleFact consists of three core components:
- Human-Adversarial Preference Testing (HAPT): Unlike synthetic data generated by other LLMs, HAPT utilizes human-crafted misinformation. Human participants write plausible but false headlines that incorporate cultural nuances and linguistic diversity, specifically designed to challenge a model's robustness against novel, non-templated attacks.
- Real-Time Web Agent with Asynchronous Validation (RTW-AV): This component ensures temporal novelty. A web agent monitors trending topics on social media platforms like X (formerly Twitter) and Instagram. It generates claims based on these emerging trends and then waits several days for professional fact-checking organizations (e.g., Snopes, PolitiFact) to release a verdict. This asynchronous approach guarantees that the ground truth was not available during the model's pre-training.
- Entity-Controlled Virtual Environment (ECVE): ECVE isolates entity-specific biases. By using Named Entity Recognition (NER), the framework replaces real-world entities with fictional analogs (e.g., replacing "Pfizer" with "VaxGen"). If a model’s accuracy significantly drops when names are changed, it indicates a reliance on memorized associations rather than semantic analysis.
The final TripleFact Score (TFS) aggregates these metrics, providing a contamination-resistant measure of a model's true detection capability.
Experimental Results
We evaluated 17 state-of-the-art LLMs, including variants of GPT-4, LLaMA-3, and DeepSeek. Our experiments involved intentional "contamination injection" to measure how traditional metrics versus the TripleFact Score respond to memorized data.
Key Findings
- The Inflation Gap: Traditional benchmarks showed performance inflation of up to 23% when models were exposed to test data. In contrast, the TFS remained remarkably stable, with an absolute error of less than 4% even under 100% contamination intensity.
- Reasoning vs. Memorization: OpenAI’s o1-preview achieved the highest TFS (0.788), demonstrating that reasoning-heavy architectures are significantly better at generalizing to new misinformation than models that rely on scale alone.
- Entity Bias: Most models showed a negative Bias Reduction Ratio (BRR), confirming that they are heavily influenced by specific entity names. Smaller models like LLaMA-3.2-1B struggled the most, highlighting a computational threshold for effective adversarial robustness.
Conclusion
TripleFact provides a necessary evolution in the evaluation of fake news detection. By neutralizing the effects of data contamination and entity bias, it offers a more honest assessment of an LLM's reasoning capabilities. As misinformation continues to evolve in real-time, frameworks that prioritize temporal novelty and adversarial robustness will be essential for maintaining trust in AI-driven fact-checking.
Summary updated