Introduction
In the current digital landscape, the rapid spread of misinformation necessitates robust automated fact-checking systems. While significant progress has been made for English, "low-resource" languages like Vietnamese often lack the high-quality, diverse datasets required to train effective models. To address this gap, researchers from the University of Information Technology in Ho Chi Minh City developed ViFactCheck, a comprehensive benchmark for multi-domain news fact-checking in Vietnamese, recently accepted at AAAI-25.
The ViFactCheck Benchmark
The primary contribution of this work is the ViFactCheck dataset, which moves beyond single-domain sources like Wikipedia to encompass real-world news across 12 diverse topics, including health, technology, education, and world politics.
The dataset consists of 7,232 human-annotated claim-evidence pairs. Unlike previous efforts that relied on translation or simple extraction, ViFactCheck utilizes human-generated claims that often require multi-step reasoning. The quality of the dataset is underscored by a Fleiss Kappa score of 0.83, indicating "very high" inter-annotator agreement.
Methodology
The creation of ViFactCheck followed a rigorous pipeline involving data collection from reputable news sources, pilot annotation, and a multi-stage validation process.
The researchers evaluated several state-of-the-art architectures:
- Fine-tuned PLMs: Models like PhoBERT (optimized for Vietnamese) and XLM-R were tested to establish a baseline.
- Large Language Models (LLMs): The study explored both zero-shot prompting and fine-tuning (via LoRA) on models such as Gemma, Mistral, Llama 3, and Gemini 1.5 Flash.
- Retrieval Systems: Evaluation included traditional BM25, semantic SBERT, and hybrid retrieval methods to identify relevant evidence within full articles.
Experimental Results
The experiments yielded several critical insights into the state of Vietnamese fact-checking:
- Superiority of Fine-tuning: Fine-tuning smaller, open-source models proved more effective than zero-shot prompting of larger models. A fine-tuned Gemma model achieved the state-of-the-art (SOTA) performance with an 89.90% macro F1 score.
- The Retrieval Bottleneck: There is a significant performance gap between models provided with "Gold Evidence" (specific relevant sentences) versus those given the "Full Context." This highlights that the primary challenge in automated fact-checking is often the retrieval of precise information from noisy sources.
- The "Refute" Challenge: Models generally excelled at identifying "Supported" claims but struggled with "Refuted" ones. This suggests that current AI architectures still find it difficult to process contradictions and negative evidence compared to simple confirmation.
- Reasoning Complexity: Performance was found to decrease as the number of evidence pieces required for a claim increased, indicating that multi-evidence synthesis remains a significant hurdle.
Conclusion and Future Work
ViFactCheck provides a vital foundation for advancing Vietnamese NLP. While the best models now approach human-level performance in certain domains, challenges remain in handling semantic ambiguity and complex logical inferences. Future research will focus on improving hybrid retrieval systems and reducing model "hallucinations" during multi-step reasoning. The researchers have open-sourced the dataset and code to encourage further development in this critical field.
Summary updated