ViFactCheck: A New Benchmark Dataset and Methods for Multi-Domain News Fact-Checking In Vietnamese

Tran Thai Hoa, Tran Quang Duy, Khanh Quoc Tran, Kiet Van NguyenAAAI 2025 · DOI

Introduction

In the current digital landscape, the rapid spread of misinformation necessitates robust automated fact-checking systems. While significant progress has been made for English, "low-resource" languages like Vietnamese often lack the high-quality, diverse datasets required to train effective models. To address this gap, researchers from the University of Information Technology in Ho Chi Minh City developed ViFactCheck, a comprehensive benchmark for multi-domain news fact-checking in Vietnamese, recently accepted at AAAI-25.

The ViFactCheck Benchmark

The primary contribution of this work is the ViFactCheck dataset, which moves beyond single-domain sources like Wikipedia to encompass real-world news across 12 diverse topics, including health, technology, education, and world politics.

Example of a claim and its supporting context in the ViFactCheck dataset, demonstrating the bilingual annotation format.

The dataset consists of 7,232 human-annotated claim-evidence pairs. Unlike previous efforts that relied on translation or simple extraction, ViFactCheck utilizes human-generated claims that often require multi-step reasoning. The quality of the dataset is underscored by a Fleiss Kappa score of 0.83, indicating "very high" inter-annotator agreement.

Methodology

The creation of ViFactCheck followed a rigorous pipeline involving data collection from reputable news sources, pilot annotation, and a multi-stage validation process.

The multi-stage annotation and validation pipeline used to ensure the high quality and reliability of the ViFactCheck benchmark.

The researchers evaluated several state-of-the-art architectures:

  1. Fine-tuned PLMs: Models like PhoBERT (optimized for Vietnamese) and XLM-R were tested to establish a baseline.
  2. Large Language Models (LLMs): The study explored both zero-shot prompting and fine-tuning (via LoRA) on models such as Gemma, Mistral, Llama 3, and Gemini 1.5 Flash.
  3. Retrieval Systems: Evaluation included traditional BM25, semantic SBERT, and hybrid retrieval methods to identify relevant evidence within full articles.

Experimental Results

The experiments yielded several critical insights into the state of Vietnamese fact-checking:

  • Superiority of Fine-tuning: Fine-tuning smaller, open-source models proved more effective than zero-shot prompting of larger models. A fine-tuned Gemma model achieved the state-of-the-art (SOTA) performance with an 89.90% macro F1 score.
  • The Retrieval Bottleneck: There is a significant performance gap between models provided with "Gold Evidence" (specific relevant sentences) versus those given the "Full Context." This highlights that the primary challenge in automated fact-checking is often the retrieval of precise information from noisy sources.
  • The "Refute" Challenge: Models generally excelled at identifying "Supported" claims but struggled with "Refuted" ones. This suggests that current AI architectures still find it difficult to process contradictions and negative evidence compared to simple confirmation.
  • Reasoning Complexity: Performance was found to decrease as the number of evidence pieces required for a claim increased, indicating that multi-evidence synthesis remains a significant hurdle.

Conclusion and Future Work

ViFactCheck provides a vital foundation for advancing Vietnamese NLP. While the best models now approach human-level performance in certain domains, challenges remain in handling semantic ambiguity and complex logical inferences. Future research will focus on improving hybrid retrieval systems and reducing model "hallucinations" during multi-step reasoning. The researchers have open-sourced the dataset and code to encourage further development in this critical field.

Summary updated

BibTeX
@inproceedings{hoa2025vifactcheck,
  author = {Tran Thai Hoa and Tran Quang Duy and Khanh Quoc Tran and Kiet Van Nguyen},
  title = {{ViFactCheck: A New Benchmark Dataset and Methods for Multi-Domain News Fact-Checking In Vietnamese}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i1.32008},
}
Figure