LiveFC: A System for Live Fact-Checking of Audio Streams

Venktesh V, Vinay Setty

TL;DR

LiveFC is a real-time automated system designed to transcribe, diarize, and fact-check live audio streams—such as political debates and rallies—within seconds. By combining specialized fine-tuned models with large language models (LLMs) for reasoning, the system identifies check-worthy claims and provides evidence-backed veracity scores faster and more accurately than general-purpose LLMs like GPT-4.

What Problem Does This Paper Address?

In the digital age, misinformation spreads like wildfire, particularly through high-engagement formats like live video and audio. While manual fact-checking by organizations like PolitiFact is the gold standard, it is slow, labor-intensive, and cannot scale to the sheer volume of live content.

Existing automated tools often suffer from two major flaws: they are "post-hoc" (operating after an event has ended) and they primarily focus on static text. This creates a dangerous gap during live events—such as election debates—where false narratives can take root in the public consciousness before a correction is ever issued. LiveFC aims to close this gap by providing a "proactive" fact-checking tool that works at the source.

Methods

The authors developed a modular pipeline that handles the messy reality of live speech:

  1. Transcription & Diarization: The system uses WhisperLive (based on OpenAI’s Whisper large-v3) for high-quality audio-to-text conversion. Simultaneously, it employs an online diarization module (based on Pyannote) to identify who is speaking, ensuring claims are attributed to the correct person in real-time.
  2. Claim Detection: A fine-tuned XLM-RoBERTa-Large model filters the transcript to identify "check-worthy" factual statements, ignoring opinions or banter.
  3. Normalization & Decomposition: Spoken language is often fragmented or context-heavy (e.g., using "he" instead of a name). The system uses Mistral-7b with Chain-of-Thought prompting to "normalize" these into self-contained questions.
  4. Evidence Retrieval: The system queries a massive index including Google, Bing, Wikipedia, and FactiSearch (a proprietary database of 280,000 existing fact-checks).
  5. Claim Verification: Finally, another fine-tuned XLM-RoBERTa-Large model performs Natural Language Inference (NLI) to determine if the retrieved evidence supports or refutes the claim.

Datasets

The researchers utilized several established benchmarks and a new internal dataset:

  • Training/Fine-tuning: ClaimBuster, CLEF CheckThat Lab!, FEVER, MNLI, and X-fact.
  • Internal Benchmark: A dataset collected from Factiverse’s production environment (1,076 training samples) was used to evaluate the claim detection and verification components.
  • Real-world Case Study: The system was tested live during the 2024 US Presidential Debate and in a pilot study with Danish fact-checkers (Tjekdet) during the European Parliament elections.

Key Insights and Ideas

  • Specialization Beats Generalization: One of the most striking findings is that specialized, smaller models (like fine-tuned XLM-RoBERTa) significantly outperformed "frontier" LLMs like GPT-4 in both claim detection and veracity prediction. The authors attribute this to LLMs' tendency to hallucinate or fail at precise information extraction from evidence snippets.
  • The "Normalization" Bridge: Converting messy live transcripts into clean, searchable questions is the "secret sauce." By resolving pronouns and dates before searching, the system drastically improves the quality of the evidence it finds.
  • Human-in-the-Loop Design: The system isn't meant to replace journalists but to augment them. By providing evidence snippets and justifications, it fosters trust and allows human fact-checkers to verify the AI's work instantly.

Key Results

  • Performance: In head-to-head tests, LiveFC’s fine-tuned model achieved a Macro-F1 score of 0.899 for claim detection, compared to just 0.695 for GPT-4.
  • Veracity Accuracy: For predicting whether a claim was true or false, LiveFC reached a 0.708 Macro-F1, outperforming GPT-4’s 0.630.
  • Debate Coverage: During the 2024 US Presidential Debate, LiveFC identified all 30 claims that PolitiFact eventually checked, while also flagging hundreds of additional claims in real-time. It processed 352 claims from Trump and 339 from Biden during the broadcast.

Limitations and Future Work

  • Input Formats: Currently, the tool is optimized for .m3u8 streams; expanding to other live formats is a priority.
  • Nuance in Labels: The current system uses a binary "Supported/Refuted" classification. The authors acknowledge the need for more nuanced labels like "Partially True" or "Conflicting."
  • Multi-modality: While it handles audio, future iterations aim to incorporate visual evidence (images/video) to detect deepfakes or misleading graphics shown during streams.
  • Language Expansion: While it supports multiple languages, the authors plan to broaden the scope to include more regional dialects and languages.

Summary updated

Abstract

The advances in the digital era have led to rapid dissemination of information. This has also aggravated the spread of misinformation and disinformation. This has potentially serious consequences, such as civil unrest. While fact-checking aims to combat this, manual fact-checking is cumbersome and not scalable. While automated fact-checking approaches exist, they do not operate in real-time and do not always account for spread of misinformation through different modalities. This is particularly important as proactive fact-checking on live streams in real-time can help people be informed of false narratives and prevent catastrophic consequences that may cause civil unrest. This is particularly relevant with the rapid dissemination of information through video on social media platforms or other streams like political rallies and debates. Hence, in this work we develop a platform named LiveFC, that can aid in fact-checking live audio streams in real-time. LiveFC has a user-friendly interface that displays the claims detected along with their veracity and evidence for live streams with associated speakers for claims from respective segments. The app can be accessed at http://livefc.factiverse.ai and a screen recording of the demo can be found at https://bit.ly/3WVAoIw.
BibTeX
@inproceedings{v2025livefc,
  author = {Venktesh V and Vinay Setty},
  title = {{LiveFC: A System for Live Fact-Checking of Audio Streams}},
  booktitle = {WSDM},
  year = {2025},
  url = {https://openreview.net/forum?id=d96qOzPS6S},
}

Loading related content...

Figure