This paper introduces TeViD (Turkish Video Fact-checking Dataset), the first multimodal video fact-checking benchmark for the Turkish language. The researchers compare two distinct architectural approaches—a zero-shot agentic pipeline and a fine-tuned "closed-book" model—to determine how well open-source large language models (LLMs) can handle misinformation in a low-resource linguistic setting.
TL;DR
The authors present TeViD, the first Turkish multimodal video fact-checking dataset, and compare an agentic, evidence-retrieving pipeline against a fine-tuned closed-book model. While the fine-tuned model achieves higher scores, analysis reveals it relies heavily on linguistic "surface signals" in the claims, whereas the agentic system demonstrates a more robust, albeit complex, verification process using live web evidence.
What Problem Does This Paper Address?
Misinformation is increasingly moving from static text to dynamic video, which is harder to verify because it requires reconciling visual, auditory, and textual information. While automated fact-checking has advanced for English, low-resource languages like Turkish lack the necessary datasets and tools.
Existing state-of-the-art systems often rely on proprietary, expensive APIs (like GPT-4o) and are designed exclusively for English. This paper tackles the challenge of building a high-quality multimodal dataset from scratch and testing whether open-source models (specifically the Qwen2.5 family) can be adapted for effective fact-checking in Turkish without relying on paid services.
Methods
The researchers developed two primary verification pipelines sharing the same open-source base models: Qwen2.5-7B (LLM) and Qwen2-VL-7B (Vision-Language Model).
- Pipeline 1: 3MFact-TR (Agentic/Open-Book): This is a zero-shot system that breaks verification into roles. It uses a Video Descriptor to turn frames into text and a Question Manager to generate search queries based on the journalistic 5W1H (Who, What, Where, When, Why, How) framework. A Bilingual Evidence Retriever then fetches Turkish and English web results (excluding known fact-checking sites to prevent data leakage) before a Reasoner produces a final verdict.
- Pipeline 2: Fine-Tuned Model (Closed-Book): This model is fine-tuned using Low-Rank Adaptation (LoRA). Unlike Pipeline 1, it does not search the web; it must decide if a claim is true or false based solely on its internal knowledge and the provided video description/transcript.
Datasets
The authors introduced TeViD, comprising 2,097 fact-checked video claims. The dataset is a hybrid of three sources:
- Teyit.org: Native Turkish claims from a certified fact-checking organization.
- Translated Snopes: English claims translated into Turkish to increase the volume of "True" examples.
- Synthetic True Examples: Real news footage from Ensonhaber paired with LLM-generated claims to further balance the dataset and prevent the model from simply guessing "False" (the majority class in native fact-checking data).
Key Insights and Ideas
- The "Surface Signal" Trap: The fine-tuned model (Pipeline 2) performed significantly better than the agentic one. However, an ablation study showed that Pipeline 2 performed almost as well when given only the text of the claim, without the video or metadata. This suggests the model learned the "writing style" of fact-checkers rather than performing actual verification.
- Balancing is Critical: Without careful balancing of the dataset (mixing native, translated, and synthetic data), models quickly learn "shortcuts"—for example, assuming any translated claim is True and any native Turkish claim is False.
- Silence vs. Contradiction: A major failure mode for the agentic pipeline was interpreting "silence" (the absence of evidence in a video) as either permission to believe a false claim or as a reason to reject a true one.
Key Results
- Fine-Tuned (P2-A): Achieved the highest performance with 0.729 macro-F1 and 0.760 accuracy.
- Agentic (Pipeline 1): Achieved 0.651 macro-F1. While lower than the fine-tuned model, it provided rationales and cited evidence, making it more explainable.
- Baseline (P0): An untrained version of the base model reached only 0.611 macro-F1, showing that both the agentic loop and fine-tuning add significant value.
- Recall: The agentic pipeline significantly improved the recall of "True" claims (0.516) compared to the untrained baseline (0.347).
Limitations and Future Work
The authors acknowledge several limitations:
- Binary Verdicts: The current system only chooses between True and False, whereas real-world fact-checking often requires an "unverifiable" or "mixed" category.
- Single-Seed Results: The fine-tuned results are based on a single training run; multi-seed replication is needed for higher statistical confidence.
- Retrieval Drift: Because Pipeline 1 relies on the live web, its results may change over time as search engine indexes evolve.
Future work includes expanding the dataset to include more fact-checking sources and refining the "abstention path" so the agentic system can explicitly state when a claim cannot be verified.
Summary updated