Turkish Multimodal Video Fact-Checking: A New Dataset and a Comparison of Zero-Shot Agentic and Fine-Tuned Systems

Azra AçılarXiv 2026 · arXiv

This paper introduces TeViD (Turkish Video Fact-checking Dataset), the first multimodal video fact-checking benchmark for the Turkish language. The researchers compare two distinct architectural approaches—a zero-shot agentic pipeline and a fine-tuned "closed-book" model—to determine how well open-source large language models (LLMs) can handle misinformation in a low-resource linguistic setting.

TL;DR

The authors present TeViD, the first Turkish multimodal video fact-checking dataset, and compare an agentic, evidence-retrieving pipeline against a fine-tuned closed-book model. While the fine-tuned model achieves higher scores, analysis reveals it relies heavily on linguistic "surface signals" in the claims, whereas the agentic system demonstrates a more robust, albeit complex, verification process using live web evidence.

What Problem Does This Paper Address?

Misinformation is increasingly moving from static text to dynamic video, which is harder to verify because it requires reconciling visual, auditory, and textual information. While automated fact-checking has advanced for English, low-resource languages like Turkish lack the necessary datasets and tools.

Existing state-of-the-art systems often rely on proprietary, expensive APIs (like GPT-4o) and are designed exclusively for English. This paper tackles the challenge of building a high-quality multimodal dataset from scratch and testing whether open-source models (specifically the Qwen2.5 family) can be adapted for effective fact-checking in Turkish without relying on paid services.

Methods

The researchers developed two primary verification pipelines sharing the same open-source base models: Qwen2.5-7B (LLM) and Qwen2-VL-7B (Vision-Language Model).

  1. Pipeline 1: 3MFact-TR (Agentic/Open-Book): This is a zero-shot system that breaks verification into roles. It uses a Video Descriptor to turn frames into text and a Question Manager to generate search queries based on the journalistic 5W1H (Who, What, Where, When, Why, How) framework. A Bilingual Evidence Retriever then fetches Turkish and English web results (excluding known fact-checking sites to prevent data leakage) before a Reasoner produces a final verdict.
  2. Pipeline 2: Fine-Tuned Model (Closed-Book): This model is fine-tuned using Low-Rank Adaptation (LoRA). Unlike Pipeline 1, it does not search the web; it must decide if a claim is true or false based solely on its internal knowledge and the provided video description/transcript.
Comparison of the agentic 3MFact-TR pipeline and the fine-tuned closed-book model.

Datasets

The authors introduced TeViD, comprising 2,097 fact-checked video claims. The dataset is a hybrid of three sources:

  • Teyit.org: Native Turkish claims from a certified fact-checking organization.
  • Translated Snopes: English claims translated into Turkish to increase the volume of "True" examples.
  • Synthetic True Examples: Real news footage from Ensonhaber paired with LLM-generated claims to further balance the dataset and prevent the model from simply guessing "False" (the majority class in native fact-checking data).

Key Insights and Ideas

  • The "Surface Signal" Trap: The fine-tuned model (Pipeline 2) performed significantly better than the agentic one. However, an ablation study showed that Pipeline 2 performed almost as well when given only the text of the claim, without the video or metadata. This suggests the model learned the "writing style" of fact-checkers rather than performing actual verification.
  • Balancing is Critical: Without careful balancing of the dataset (mixing native, translated, and synthetic data), models quickly learn "shortcuts"—for example, assuming any translated claim is True and any native Turkish claim is False.
  • Silence vs. Contradiction: A major failure mode for the agentic pipeline was interpreting "silence" (the absence of evidence in a video) as either permission to believe a false claim or as a reason to reject a true one.
Worked examples showing how the pipelines handle location lies and causal claims.

Key Results

  • Fine-Tuned (P2-A): Achieved the highest performance with 0.729 macro-F1 and 0.760 accuracy.
  • Agentic (Pipeline 1): Achieved 0.651 macro-F1. While lower than the fine-tuned model, it provided rationales and cited evidence, making it more explainable.
  • Baseline (P0): An untrained version of the base model reached only 0.611 macro-F1, showing that both the agentic loop and fine-tuning add significant value.
  • Recall: The agentic pipeline significantly improved the recall of "True" claims (0.516) compared to the untrained baseline (0.347).

Limitations and Future Work

The authors acknowledge several limitations:

  • Binary Verdicts: The current system only chooses between True and False, whereas real-world fact-checking often requires an "unverifiable" or "mixed" category.
  • Single-Seed Results: The fine-tuned results are based on a single training run; multi-seed replication is needed for higher statistical confidence.
  • Retrieval Drift: Because Pipeline 1 relies on the live web, its results may change over time as search engine indexes evolve.

Future work includes expanding the dataset to include more fact-checking sources and refining the "abstention path" so the agentic system can explicitly state when a claim cannot be verified.

Summary updated

Abstract

Video-based misinformation spreads rapidly on social media, but almost all existing fact-checking systems focus exclusively on English text. Tools for low-resource languages such as Turkish are severely limited. In this work, we present TeViD (Turkish Video Fact-checking Dataset), to the best of our knowledge the first multimodal video fact-checking dataset for Turkish. We compiled the dataset from articles published by Teyit.org, an IFCN-certified organisation, and enriched it with translated Snopes articles and synthetic examples to reduce class imbalance. Building on TeViD, we develop two verification pipelines that share the same open-source base model. First, in 3MFact-TR (Pipeline 1), we adapt the multi-role 3MFact framework to Turkish with the Qwen2.5-7B large language model and the Qwen2-VL-7B vision-language model, and add three modules of our own: a semantic inconsistency checker, a 5W1H question manager, and a bilingual evidence retriever that excludes fact-checking sites to prevent leakage. Second, as our Fine-Tuned Model (Pipeline 2), we fine-tune a single Qwen2.5-7B with Low-Rank Adaptation (LoRA) to produce closed-book verdicts from the claim and the video's textual description. We report accuracy, macro F1, True class F1, and True class recall for both systems, together with ablation studies.
BibTeX
@misc{acl2026turkish,
  author = {Azra Açıl},
  title = {{Turkish Multimodal Video Fact-Checking: A New Dataset and a Comparison of Zero-Shot Agentic and Fine-Tuned Systems}},
  year = {2026},
  eprint = {2611.08651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08651},
}
Figure