Abstract
This paper investigates the potential benefits of language-specific fact-checking models, focusing on the case of Chinese. We first demonstrate the limitations of translation-based methods and multilingual large language models (e.g., GPT-4), highlighting the need for language-specific systems. We further propose a Chinese fact-checking system that can better retrieve evidence from a document by incorporating context information. To better analyze token-level biases in different systems, we construct an adversarial dataset based on the CHEF dataset, where each instance has large word overlap with the original one but holds the opposite veracity label. Experimental results on the CHEF dataset and our adversarial dataset show that our proposed method outperforms translation-based methods and multilingual LLMs and is more robust toward biases, while there is still large room for improvement, emphasizing the importance of language-specific fact-checking systems.
Introduction
Automated fact-checking has historically been an English-centric field. As researchers look to expand these tools to other languages, a fundamental question arises: Do we need to build bespoke models for every language, or can we rely on "translation-plus-English-model" pipelines and massive multilingual LLMs like GPT-4?
This research identifies two major hurdles for the "one-size-fits-all" approach. First, linguistic nuance is often lost; translation frequently fails to capture idiomatic expressions or specific technical jargon, leading to errors that flip the meaning of a claim. Second, cultural and dataset biases are prevalent. Models often learn statistical shortcuts (e.g., "if it's about a virus, it's probably fake") rather than performing actual reasoning.
Methodology
The researchers developed a comprehensive pipeline to test the necessity of language-specific models:
- Document-level Retriever (DLR): Unlike previous models that examine sentences in isolation, the DLR utilizes a BigBird architecture to consider the entire context of a document. It assigns scores to individual tokens and aggregates them to identify the most relevant evidence sentences.
- Adversarial Dataset Construction: To evaluate whether models were performing factual reasoning or merely matching keywords, the authors used GPT-4 to create a "symmetrical" adversarial dataset. They modified original claims (altering dates, names, or numbers) to create new claims with the opposite truth value but high word overlap with the original.
- Inoculation by Fine-Tuning: The study tested if exposing models to a small amount of this adversarial data could help them overcome reliance on surface-level patterns.
Datasets
The study utilizes two primary datasets:
- CHEF: A real-world Chinese evidence-based fact-checking dataset used for training and baseline evaluation.
- Adversarial CHEF: A new benchmark consisting of 1,000 claim-evidence pairs (250 original, 750 generated). This dataset is specifically designed to challenge models that rely on keyword matching rather than reasoning.
Results and Analysis
Performance on the CHEF Dataset
The proposed system, combining the DLR with a Chinese-specific DeBERTa model, achieved state-of-the-art performance. As shown in the experimental results, the language-specific approach significantly outperforms both translation-based methods and general multilingual LLMs.
Specifically, the DLR + Chinese DeBERTa configuration reached 74.50% accuracy and Macro F1, outperforming the best translation-based methods by over 10%. The DLR itself improved evidence retrieval accuracy by approximately 5% over standard semantic rankers.
The Robustness Gap
While language-specific models perform well on standard datasets, the adversarial evaluation reveals a significant "robustness gap." When tested on the Adversarial CHEF dataset, all models experienced a massive performance drop.
For instance, Chinese DeBERTa’s accuracy fell from 86.69% on original pairs to 57.84% on generated adversarial ones. This suggests that even the most advanced models still rely heavily on dataset biases and statistical shortcuts.
Insights on Bias and LLMs
The study highlights several critical issues:
- Statistical Shortcuts: In the CHEF dataset, significant domain bias exists. For instance, 66% of health claims are "REFUTED," while 72% of culture claims are "SUPPORTED." Models often exploit these ratios instead of verifying evidence.
- Western Norms in LLMs: Multilingual LLMs like ChatGPT often reflect Western values, making them less effective for claims rooted in Chinese social or political contexts.
- Architecture and Learning: During inoculation tests, advanced models like DeBERTa showed a better ability to learn from adversarial examples compared to simpler models like BERT, indicating that architecture is key to handling linguistic complexity.
Conclusion
This research demonstrates that language-specific models are essential for effective fact-checking in non-English contexts. While translation-based and general multilingual models offer a starting point, they struggle with linguistic nuances and deep-seated biases. The path forward involves building more culturally and linguistically specialized models that can move beyond statistical shortcuts toward true factual reasoning. Future work will expand this methodology to other languages to build a more robust, globally-aware fact-checking infrastructure.
Summary updated