TL;DR
This paper introduces TSVer, a novel benchmark dataset for fact verification that specifically targets the challenging domain of time-series evidence. It addresses the limitations of existing datasets by providing real-world claims, rich time-series data, and detailed, structured justifications, pushing the boundaries of how AI systems reason with temporal and numerical information.
What Problem Does This Paper Address?
Fact-checking is becoming increasingly critical in our information-saturated world, especially with the rise of generative AI. While automated fact-checking systems have advanced, they often struggle with claims that require reasoning over temporal and numerical data, particularly when the evidence is in the form of time series. Existing datasets either lack structured evidence, provide insufficient justifications, or use synthetic claims, making it difficult to accurately evaluate a system's ability to handle this complex type of information. This gap hinders the development of robust fact-checking AI that can reliably assess claims involving trends, historical data, and evolving facts.
Methods
The authors developed a multi-stage annotation process to create the TSVer dataset. This involved:
- Claim Extraction: Sourcing real-world claims from fact-checking organizations and filtering them for those involving temporal and numerical expressions.
- Time-Series Data Curation: Collecting and preprocessing a large dataset of time series from "Our World in Data," covering diverse domains and featuring extensive historical data.
- LLM-Assisted Annotation: Employing a two-round annotation study. Phase 1 involved human annotators selecting relevant time series and time frames. Phase 2 involved human annotators assigning verdicts and generating justifications, with the aid of GPT-4 to generate initial factual assertions that were then refined by annotators. This process aimed to improve the quality and detail of justifications.
- Baseline Pipeline: Developing a baseline system that uses LLMs for both time-series retrieval and verdict/justification generation.
Datasets
- TSVer Dataset: The core contribution is this newly introduced benchmark dataset. It comprises:
- 287 real-world claims sourced from 38 fact-checking organizations.
- 400 curated time series from "Our World in Data," with an average of 20,000 records per instance, some exceeding 217,000 records.
- Annotations include time frames, verdicts (SUPPORTS, REFUTES, NOT ENOUGH INFO, CONFLICTING EVIDENCE/CHERRY-PICKING), and detailed, structured justifications.
- Existing Datasets: The paper compares TSVer to existing fact-checking datasets like FEVER, FEVEROUS, TabFact, SciTab, FinDVer, and QuanTemp, highlighting TSVer's unique focus on time-series evidence and structured justifications.
Key Insights and Ideas
- Time Series are a Hard Frontier: The results clearly show that even state-of-the-art LLMs struggle significantly with time-series fact verification, indicating this is a challenging area for current AI.
- Justification Quality Matters: The Ev²R score, which measures factual alignment in justifications, proved to be a more informative metric than surface-level metrics like METEOR, highlighting the importance of deep reasoning in explanations.
- Cherry-Picking is a Real Challenge: The dataset's design explicitly accounts for the practice of cherry-picking by including claims where specific date ranges can lead to different verdicts, mirroring real-world misinformation tactics.
- Data Volume and Complexity: The sheer size and temporal nature of the time-series data in TSVer present unique challenges for LLMs, particularly concerning context window limitations and efficient data processing.
Key Results
- Verdict Accuracy: State-of-the-art models like Gemini-2.5-Pro achieved only 63.37% accuracy on verdict prediction, and GPT-4 achieved 65.35%. Smaller open-weight models performed much worse, with Llama-3.1-8B scoring as low as 7.92%.
- Retrieval Performance: Gemini achieved the highest Time Series Coverage Score (TSCS) of 41.39, indicating better ability to select relevant time series and align time ranges compared to other models.
- Justification Quality: Gemini also led in the Ev²R score for justifications with 48.63, demonstrating superior factual alignment in its explanations.
- Inter-Annotator Agreement: The dataset achieved a substantial inter-annotator agreement of κ = 0.745 for verdicts, indicating high consistency among human annotators.
Limitations and Future Work
- Multilingual Coverage: The current dataset is limited to English claims and articles, preventing assessment of cross-lingual or multilingual reasoning capabilities.
- Evidence Modality: TSVer exclusively uses time-series evidence. Real-world claims often involve multiple evidence types (text, tables, images, etc.), and future work could explore multi-modal fact verification.
- Representation of Time Series: The paper notes that representing time series as raw text for LLMs presents challenges due to input length and tokenization issues. Future work could explore more efficient time-series representations or specialized encoders.
- Model Limitations: The authors suggest that exploring more efficient time-series representations might enable future systems to better encode and reason over temporal data within constrained model budgets.
Summary updated