TL;DR
LiveFC is a real-time automated system designed to transcribe, diarize, and fact-check live audio streams—such as political debates and rallies—within seconds. By combining specialized fine-tuned models with large language models (LLMs) for reasoning, the system identifies check-worthy claims and provides evidence-backed veracity scores faster and more accurately than general-purpose LLMs like GPT-4.
What Problem Does This Paper Address?
In the digital age, misinformation spreads like wildfire, particularly through high-engagement formats like live video and audio. While manual fact-checking by organizations like PolitiFact is the gold standard, it is slow, labor-intensive, and cannot scale to the sheer volume of live content.
Existing automated tools often suffer from two major flaws: they are "post-hoc" (operating after an event has ended) and they primarily focus on static text. This creates a dangerous gap during live events—such as election debates—where false narratives can take root in the public consciousness before a correction is ever issued. LiveFC aims to close this gap by providing a "proactive" fact-checking tool that works at the source.
Methods
The authors developed a modular pipeline that handles the messy reality of live speech:
- Transcription & Diarization: The system uses WhisperLive (based on OpenAI’s Whisper large-v3) for high-quality audio-to-text conversion. Simultaneously, it employs an online diarization module (based on Pyannote) to identify who is speaking, ensuring claims are attributed to the correct person in real-time.
- Claim Detection: A fine-tuned XLM-RoBERTa-Large model filters the transcript to identify "check-worthy" factual statements, ignoring opinions or banter.
- Normalization & Decomposition: Spoken language is often fragmented or context-heavy (e.g., using "he" instead of a name). The system uses Mistral-7b with Chain-of-Thought prompting to "normalize" these into self-contained questions.
- Evidence Retrieval: The system queries a massive index including Google, Bing, Wikipedia, and FactiSearch (a proprietary database of 280,000 existing fact-checks).
- Claim Verification: Finally, another fine-tuned XLM-RoBERTa-Large model performs Natural Language Inference (NLI) to determine if the retrieved evidence supports or refutes the claim.
Datasets
The researchers utilized several established benchmarks and a new internal dataset:
- Training/Fine-tuning: ClaimBuster, CLEF CheckThat Lab!, FEVER, MNLI, and X-fact.
- Internal Benchmark: A dataset collected from Factiverse’s production environment (1,076 training samples) was used to evaluate the claim detection and verification components.
- Real-world Case Study: The system was tested live during the 2024 US Presidential Debate and in a pilot study with Danish fact-checkers (Tjekdet) during the European Parliament elections.
Key Insights and Ideas
- Specialization Beats Generalization: One of the most striking findings is that specialized, smaller models (like fine-tuned XLM-RoBERTa) significantly outperformed "frontier" LLMs like GPT-4 in both claim detection and veracity prediction. The authors attribute this to LLMs' tendency to hallucinate or fail at precise information extraction from evidence snippets.
- The "Normalization" Bridge: Converting messy live transcripts into clean, searchable questions is the "secret sauce." By resolving pronouns and dates before searching, the system drastically improves the quality of the evidence it finds.
- Human-in-the-Loop Design: The system isn't meant to replace journalists but to augment them. By providing evidence snippets and justifications, it fosters trust and allows human fact-checkers to verify the AI's work instantly.
Key Results
- Performance: In head-to-head tests, LiveFC’s fine-tuned model achieved a Macro-F1 score of 0.899 for claim detection, compared to just 0.695 for GPT-4.
- Veracity Accuracy: For predicting whether a claim was true or false, LiveFC reached a 0.708 Macro-F1, outperforming GPT-4’s 0.630.
- Debate Coverage: During the 2024 US Presidential Debate, LiveFC identified all 30 claims that PolitiFact eventually checked, while also flagging hundreds of additional claims in real-time. It processed 352 claims from Trump and 339 from Biden during the broadcast.
Limitations and Future Work
- Input Formats: Currently, the tool is optimized for
.m3u8streams; expanding to other live formats is a priority. - Nuance in Labels: The current system uses a binary "Supported/Refuted" classification. The authors acknowledge the need for more nuanced labels like "Partially True" or "Conflicting."
- Multi-modality: While it handles audio, future iterations aim to incorporate visual evidence (images/video) to detect deepfakes or misleading graphics shown during streams.
- Language Expansion: While it supports multiple languages, the authors plan to broaden the scope to include more regional dialects and languages.
Summary updated