Abstract
The advances in the digital era have led to rapid dissemination of information, aggravating the spread of misinformation and disinformation with potentially serious consequences, such as civil unrest. While fact-checking aims to combat this, manual processes are cumbersome and not scalable. Existing automated approaches often fail to operate in real-time or account for multi-modal misinformation. We present LiveFC, a platform designed to fact-check live audio streams in real-time. LiveFC features a user-friendly interface that displays detected claims, their veracity, and supporting evidence, attributed to specific speakers. The system is accessible at http://livefc.factiverse.ai.
Introduction
In the digital age, misinformation spreads rapidly through high-engagement formats like live video and audio. While manual fact-checking by organizations like PolitiFact remains the gold standard, it is labor-intensive and cannot scale to the volume of live content. Existing automated tools are often "post-hoc," operating only after an event concludes. This creates a dangerous gap during live events—such as election debates—where false narratives can take root before corrections are issued. LiveFC aims to close this gap by providing a proactive, real-time fact-checking tool.
System Architecture and Methods
LiveFC utilizes a modular pipeline designed to handle the complexities of live speech, as illustrated in the system architecture below:
- Transcription & Diarization: The system uses WhisperLive (OpenAI’s Whisper large-v3) for audio-to-text conversion. Simultaneously, an online diarization module based on Pyannote identifies speakers to ensure claims are correctly attributed.
- Claim Detection: A fine-tuned XLM-RoBERTa-Large model filters the transcript to identify "check-worthy" factual statements.
- Normalization & Decomposition: To handle fragmented spoken language, Mistral-7b uses Chain-of-Thought prompting to normalize claims into self-contained, searchable questions.
- Evidence Retrieval: The system queries a massive index including Google, Bing, Wikipedia, and FactiSearch, a proprietary database of 280,000 existing fact-checks.
- Claim Verification: A second fine-tuned XLM-RoBERTa-Large model performs Natural Language Inference (NLI) to determine if retrieved evidence supports or refutes the claim.
Experimental Results and Case Study
LiveFC was evaluated against established benchmarks (ClaimBuster, FEVER, etc.) and a real-world case study during the 2024 US Presidential Debate.
Specialized Models vs. General LLMs
A key finding of this work is that specialized, smaller models significantly outperform general-purpose large language models (LLMs). In head-to-head tests, LiveFC’s fine-tuned model achieved a Macro-F1 score of 0.899 for claim detection, compared to 0.695 for GPT-4. For veracity prediction, LiveFC reached a 0.708 Macro-F1, outperforming GPT-4’s 0.630. This suggests that LLMs may struggle with precise information extraction from evidence snippets or suffer from hallucinations in this specific domain.
Real-World Deployment
During the 2024 US Presidential Debate, LiveFC identified all 30 claims eventually checked by PolitiFact, while flagging hundreds of additional claims in real-time. The system processed 352 claims from Donald Trump and 339 from Joe Biden during the broadcast. The interface provides immediate visual feedback on the veracity of these claims.
Discussion and Future Work
LiveFC demonstrates the efficacy of a "Human-in-the-Loop" design, augmenting journalists by providing instant evidence snippets and justifications. However, several areas for improvement remain:
- Nuanced Labeling: Moving beyond binary "Supported/Refuted" labels to include "Partially True" or "Conflicting."
- Multi-modality: Incorporating visual evidence to detect deepfakes or misleading graphics.
- Language Support: Expanding beyond current capabilities to include more regional dialects and a wider array of global languages.
By providing real-time, evidence-backed verification, LiveFC serves as a critical tool in preventing the entrenchment of false narratives during high-stakes live events.
Summary updated