TL;DR
This paper introduces CFEVER, the largest Chinese dataset for fact extraction and verification, built upon Chinese Wikipedia. It addresses the critical need for robust fact-checking tools in Chinese, a language often characterized by ambiguity, and provides a challenging benchmark for developing such systems.
What Problem Does This Paper Address?
The rapid spread of misinformation across media platforms necessitates automated fact-verification systems. While significant progress has been made in English, the landscape for Chinese fact verification is less developed. Chinese text, with its inherent ambiguity and nuance, poses unique challenges for identifying and verifying factual claims. Existing Chinese datasets are often limited in size or focus solely on rumor detection, lacking the comprehensive scope of evidence extraction and verification found in English benchmarks like FEVER. This paper aims to bridge this gap by creating a large-scale, high-quality dataset specifically for Chinese fact extraction and verification.
Methods
The authors followed a methodology inspired by the FEVER dataset. They first constructed a fact database using a fixed version of the Chinese Wikipedia dump. Human workers, recruited from their university, then generated claims based on sentences from this database. These claims were labeled as "Supports," "Refutes," or "Not Enough Info." For claims labeled "Supports" or "Refutes," corresponding evidence sentences were also identified from the Wikipedia pages. The dataset was rigorously validated, achieving a Fleiss' kappa score of 0.7934 for inter-annotator agreement, indicating high label consistency. The paper also details experiments using baseline systems and the state-of-the-art BEVERS model to evaluate the dataset's difficulty.
Datasets
The primary contribution is the CFEVER dataset, comprising 30,012 manually created claims based on Chinese Wikipedia content. This dataset is significantly larger than existing Chinese fact-checking datasets like CHEF. CFEVER is split into training, development, and test sets. The authors also reference and compare their work against the FEVER dataset and its variations, as well as other Chinese datasets like Weibo-16 and Weibo-20.
Key Insights and Ideas
- The need for Chinese-specific benchmarks: The inherent linguistic differences between Chinese and English make direct application of English-developed models challenging, highlighting the importance of dedicated Chinese datasets.
- Wikipedia as a rich source: Leveraging Wikipedia, a widely accessible and comprehensive knowledge base, proves effective for constructing a large-scale fact verification dataset.
- Inter-annotator agreement is crucial: A high Fleiss' kappa score (0.7934) demonstrates the reliability and quality of the annotations, a critical factor for training robust models.
- Multi-hop reasoning remains a challenge: While CFEVER is designed to be more realistic than some prior datasets, the analysis suggests that claims requiring evidence from multiple pages are still difficult for current models.
Key Results
- The CFEVER dataset contains 30,012 claims, making it the largest Chinese dataset for evidence-based fact verification.
- The dataset achieved a high inter-annotator agreement of 0.7934 (Fleiss' kappa), surpassing scores from FEVER and CHEF.
- Experiments with baseline and state-of-the-art models (BEVERS) showed that CFEVER presents a significant challenge, with BEVERS achieving a FEVER score of 64.80% on the full pipeline, indicating room for improvement.
- Performance degrades significantly for claims requiring evidence from multiple Wikipedia pages, suggesting that multi-hop reasoning is a key area for future research.
Limitations and Future Work
The authors acknowledge two main limitations:
- Dataset Size: While CFEVER is the largest Chinese dataset, it is still smaller than the English FEVER dataset. They aim to increase the data scale in the future while maintaining annotation quality.
- Complex Reasoning: Most claims in CFEVER require evidence from a single Wikipedia page, similar to FEVER. The authors hope to expand the dataset with more complex claims that necessitate evidence from multiple pages to better support multi-step reasoning tasks.
Summary updated