Abstract
This paper investigates the potential benefits of language-specific fact-checking models, focusing on the case of Chinese using the CHEF dataset. To better reflect real-world fact-checking, we first develop a novel Chinese document-level evidence retriever, achieving state-of-the-art performance. We then demonstrate the limitations of translation-based methods and multilingual language models, highlighting the need for language-specific systems. To better analyze token-level biases in different systems, we construct an adversarial dataset based on the CHEF dataset, where each instance has a large word overlap with the original one but holds the opposite veracity label. Experimental results on the CHEF dataset and our adversarial dataset show that our proposed method outperforms translation-based methods and multilingual language models and is more robust toward biases, emphasizing the importance of language-specific fact-checking systems.
Introduction
Automated fact-checking has become a critical tool in combating global misinformation. However, the field remains heavily "English-centric," with most datasets and models optimized for Western linguistic and cultural contexts. When addressing misinformation in other languages, such as Mandarin Chinese, researchers often rely on two common shortcuts: "Translate-then-Verify" (translating content into English for processing by English models) or utilizing massive multilingual Large Language Models (LLMs) like GPT-4.
This study argues that these shortcuts are insufficient and potentially dangerous for high-stakes tasks. Translation often fails to capture idiomatic nuances, while general multilingual models frequently carry Western-centric biases that do not align with regional information ecosystems. We propose that language-specific models are essential for navigating these unique linguistic and cultural complexities.
Methodology
We developed a complete fact-checking pipeline consisting of two primary stages: retrieval and verification.
Document-Level Retrieval (DLR)
Unlike traditional sentence-level rankers, our Document-Level Retriever (DLR) utilizes a BigBird architecture. This allows the model to process entire documents up to 4,096 tokens, which is vital for Chinese evidence documents that average 866 tokens—well beyond the capacity of standard BERT-based models. The DLR identifies relevant evidence by aggregating token-level scores across the full document context.
Verifier Models
We compared three categories of verifiers to determine the veracity of claims (Supported, Refuted, or Not Enough Info):
- Translation-based: Utilizing Google Translate or GPT-4 to convert claims to English, followed by an English DeBERTa verifier.
- Multilingual: Using mDeBERTa and GPT-3.5/4-Turbo directly on the Chinese text.
- Chinese-Specific: Models pre-trained specifically on Chinese corpora, such as Chinese RoBERTa and DeBERTa.
Adversarial Benchmark
To test if models were truly reasoning or merely performing keyword matching, we created an Adversarial CHEF dataset. Using GPT-4, we generated "symmetrical" claims—altering a claim just enough to flip its truth value while maintaining high word overlap with the original. This exposes whether a model relies on statistical biases (e.g., assuming any claim about a certain topic is always "Refuted").
Experimental Results
Performance on CHEF
Our results demonstrate the clear superiority of native Chinese models. The pipeline combining our DLR with Chinese DeBERTa achieved the highest performance, reaching 74.50% accuracy. In contrast, the best translation-based method (GPT-4 + DeBERTa) lagged behind by over 10% in accuracy. Multilingual LLMs also struggled, often failing to capture the specific surface-level patterns inherent in Chinese news and social media.
Robustness and Adversarial Analysis
The adversarial testing revealed significant brittleness across all models. When tested on the generated adversarial pairs, performance plummeted. For instance, Chinese DeBERTa's accuracy dropped from approximately 86% on original pairs to 57% on the adversarial subset. This indicates that even state-of-the-art models still over-rely on linguistic cues and cultural "shortcuts" rather than deep factual reasoning.
Discussion: The Translation and Bias Gap
The "translation gap" was evident in several cases where idiomatic Chinese phrases were misinterpreted during translation, leading to incorrect verifications. Furthermore, we identified significant statistical biases in the dataset; claims mentioning "China" were statistically more likely to be labeled Supported, while those mentioning "Virus" or "Taiwan" were frequently Refuted. Language-specific models proved more capable of navigating these biases, though they are not yet immune to them.
Conclusion
Our findings emphasize that language-specific fact-checking systems are indispensable for non-English contexts. While translation and multilingual LLMs offer convenience, they lack the precision and cultural grounding necessary for reliable fact-checking. Future work will focus on improving retrieval recall and expanding these methodologies to other languages to build more robust global misinformation defenses.
Summary updated