TL;DR
This paper introduces the Semantic Sensitivity Amplifier (SSA), a novel framework to detect "semantic contamination" in Large Language Models (LLMs) used for fake news detection. By systematically swapping named entities within news statements, SSA quantifies how much an LLM's predictions rely on memorized knowledge rather than genuine reasoning, revealing hidden biases and inflated performance metrics.
What Problem Does This Paper Address?
The rapid advancement of LLMs has revolutionized Natural Language Processing, but their evaluation is increasingly plagued by Benchmark Data Contamination (BDC). While previous work focused on direct data or label overlap, this paper highlights a more insidious form: semantic contamination. This occurs when LLMs inadvertently learn subtle conceptual similarities or factual knowledge from their training data that directly overlap with evaluation benchmarks. In fake news detection, this is particularly problematic, as it can lead models to "remember" answers instead of truly reasoning about the truthfulness of a claim, leading to deceptively high accuracy scores and undermining the reliability of LLM evaluations. The authors aim to formally define and address this semantic BDC.
Methods
The core of the proposed solution is the Semantic Sensitivity Amplifier (SSA) framework. It operates in two main stages:
- Entity Shift: This is the novel perturbation technique. Key named entities (people, organizations, locations) within a news statement are identified and systematically replaced with neutral or fictional entities. Crucially, the truthfulness of the statement remains constant, and auxiliary context is provided to maintain semantic coherence. This process is designed to test if the LLM's prediction is invariant to changes in specific entities, which it should be if it's reasoning based on content rather than memorized associations.
- SSA Factor Calculation: After applying the entity shift to create a "revised benchmark," the LLM is evaluated on both the original and revised datasets. Two metrics are computed:
- Accuracy Difference (Δ): The change in accuracy between the original and shifted datasets. A large negative Δ indicates performance degradation due to the entity shift.
- Overturn Rate (OTR): The fraction of individual predictions that flip after the entity shift. A high OTR suggests sensitivity to entity changes. The SSA Factor is then calculated as the product of Δ and OTR (multiplied by 100). This composite metric provides a sensitive and interpretable indicator of semantic contamination risk.
Datasets
- LIAR2 Dataset: Used for evaluating LLMs on fake news detection. It's a benchmark of approximately 23,000 manually labeled short statements.
- RefinedWeb: Used as a source for general language diversity in the contaminated corpus.
- OpenWebText: Used as a validation set during the pre-training process to monitor for overfitting and ensure preserved model capabilities.
The paper also describes the process of simulating semantic contamination by creating "near-domain" and "information-level" corpora, and directly using LIAR2's training set for data/label level contamination.
Key Insights and Ideas
- Semantic BDC is a critical blind spot: Traditional BDC detection methods are insufficient for subtle semantic overlaps, which can significantly inflate LLM performance without reflecting true reasoning abilities.
- Entity invariance is key to robust reasoning: A model that truly understands fake news detection should not change its prediction based solely on swapping named entities, as long as the core claim and its truthfulness remain the same.
- SSA is a sensitive and scalable audit: The SSA Factor effectively captures semantic contamination, showing a strong correlation with injected BDC levels, even for subtle forms.
- Larger models are more vulnerable (and reveal more): Larger LLMs, with their enhanced memorization capabilities, tend to exhibit higher SSA Factors when contaminated, making SSA a particularly useful tool for auditing state-of-the-art models.
- SSA can be a dual-use tool: Beyond detection, the entity shift process itself can act as a form of mitigation by disrupting reliance on memorized knowledge.
Key Results
- Monotonic performance inflation: LLM accuracy on the LIAR2 dataset consistently increased with the severity of injected contamination across all tested models and BDC levels.
- SSA Factor tracks BDC risk: The SSA Factor showed a near-perfect lock-step escalation with injected contamination (r ≥ .97 for models ≥ 3B parameters).
- Scalability demonstrated: The framework was tested on 45 LLM variants ranging from 0.5B to 72B parameters, demonstrating its applicability across a wide spectrum of model sizes.
- Model-specific sensitivities: Larger Qwen2.5 models (≥ 7B parameters) showed substantially higher baseline accuracies and larger SSA Factors compared to smaller models, indicating their greater susceptibility to contamination.
- Cost-effectiveness: The SSA framework is computationally lightweight, requiring only running the model on original and perturbed inputs, making it feasible for large-scale evaluations.
Limitations and Future Work
- Prohibitive cost of large-scale pre-training: The study was limited to models up to 72B parameters due to the expense of continued pre-training. Evaluating even larger models (e.g., Llama-3.1-405B) remains an open question.
- Limited open-corpus training: Only two checkpoints were trained from scratch on fully open corpora, limiting conclusions about how corpus choice alone modulates semantic contamination.
- Simulated contamination pipeline: The contamination injection was simulated in a single pass, which might differ from real-world industrial training schedules.
- Inherent flaw in entity shift: The authors acknowledge that the entity shift process might subtly alter semantics or introduce new biases, despite efforts to mitigate this through manual review.
- Future work: The authors suggest extending the approach to longer, multimodal contexts and developing fully automated perturbation pipelines. They also propose that SSA can be repurposed as a mitigation strategy.
Summary updated