Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses

Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi, Ping WangarXiv 2026 · arXiv

TL;DR

LLMs often provide confident answers to population-level clinical questions without actually performing the necessary statistical analysis, leading to a "verdict-evidence gap." This paper introduces STRATA, a benchmark that evaluates every step of the hypothesis-testing process, and ART (Articulate, Retrieve, Test), a staged pipeline that significantly improves the reliability and transparency of temporal reasoning in clinical data.

What Problem Does This Paper Address?

When you ask an AI, "Does symptom A usually follow vaccine B?" it often responds with a fluent, authoritative sentence. However, in high-stakes domains like healthcare, a "correct" answer is meaningless if it isn't grounded in a rigorous statistical test over the right data.

The authors identify a verdict-evidence gap: existing benchmarks only grade the final answer, ignoring whether the model correctly identified the null hypothesis, selected the right statistical test, or even counted the correct records. A model might get the right verdict by sheer luck or by using a flawed SQL query that happens to return a similar count. This lack of "traceability" makes LLMs dangerous for clinical discovery.

A worked example showing how a single-query baseline can get the right verdict for the wrong reasons (the verdict-evidence gap).

Methods

The authors propose a two-part solution: a new benchmark and a new reasoning paradigm.

1. The STRATA Benchmark

STRATA (STatistical Reasoning Assessment for Temporal Assertions) is the first benchmark to provide "gold" labels for every stage of the analysis. It includes 8,735 hypotheses generated from clinical timelines. For every hypothesis, STRATA provides:

  • The Plan: The specific null hypothesis, statistical test (e.g., exact binomial), and conditioning rules.
  • The Evidence: The exact set of report IDs that should be included in the analysis.
  • The Statistics: The gold $p$-values and final verdicts (supported, refuted, or insufficient).

2. The ART Pipeline

To solve the task, the authors introduce ART (Articulate, Retrieve, Test), a staged verification paradigm:

  • S1: Articulate: The LLM generates a structured JSON plan naming the events, the null hypothesis, and the required statistical test.
  • S2: Retrieve: The model writes SQL queries to fetch the evidence. A "contract" harness ensures these queries follow the rules defined in the plan.
  • S3: Test: A deterministic function (not an LLM) runs the actual statistical test on the retrieved counts to produce the final verdict.

The pipeline also includes a verify-and-repair loop, where the system checks the plan and retrieved data for consistency against a provided "codebook" of rules.

Figure 2: Overview of the Strata benchmark and the Art pipeline: (1) construction, (2) verification, (3) the measure that scores each stage against its gold.

Datasets

  • MedTempo: The benchmark is built on clinical timelines extracted from 3,166 VAERS (Vaccine Adverse Event Reporting System) reports related to COVID-19 vaccines.
  • STRATA Pool: 8,735 hypotheses across 26 template groups, categorized into three difficulty levels (Easy, Medium, Hard) and seven hypothesis families (e.g., precedes, co-occurs, proximity).
  • Evaluation Set: A 1,500-hypothesis test set balanced across verdicts.

Key Insights and Ideas

  • The "Right for the Wrong Reasons" Problem: Experiments showed that without a staged pipeline, models frequently fail at choosing which records count. For example, a model might correctly guess that "Pain precedes Asthenia" but its underlying query might have accidentally excluded all cases where Asthenia came first.
  • Codebooks are Essential: Frontier models (like GPT-5.6 or Claude 5.5) are highly capable but require an explicit "codebook"—a text-based set of rules defining how to interpret temporal language—to reach near-perfect accuracy.
  • Staging Levels the Playing Field: By breaking the task into Articulate, Retrieve, and Test, smaller open-source models (like a 31B Gemma 4) can perform nearly as well as massive frontier models. The bottleneck for smaller models is usually the complex logic of the "Retrieve" (SQL) stage, which the ART pipeline helps manage.

Key Results

  • Accuracy Boost: The ART pipeline improved verdict accuracy by +0.23 for Qwen2.5-32B and +0.12 for Gemma4 compared to standard single-query baselines.
  • Closing the Gap: Under the ART paradigm, the 31B Gemma 4 model achieved an accuracy of 0.981, coming within 0.01 of the best frontier models.
  • Traceability: ART significantly raised "faithfulness" scores (whether the verdict rests on the correct evidence). For open models, faithfulness rose from as low as 0.26 to over 0.86.
Analysis of why models fail to identify "insufficient" data, showing that overclaiming is a major issue for baselines.

Limitations and Future Work

  • Corpus Specificity: The current codebook and primitives are specific to the MedTempo/VAERS dataset. Extending STRATA to new clinical domains would require writing new domain-specific rules.
  • Template Dependency: The hypotheses are currently generated via templates. The authors suggest that testing against human-written hypotheses would be a valuable next step to see if models can map natural language to the correct statistical plans without template-based cues.
  • Harness Complexity: The "contract" harness that expands primitives into SQL is currently hand-written, which may limit the system's flexibility in entirely new data environments.

Summary updated

Abstract

Ask a language model whether sales tend to dip after a product recall, or whether one side effect tends to follow another after a vaccine, and it answers in a fluent sentence that reads like a finding, with no test behind it. Assertions about how events unfold across many cases fill the Web, and each is a temporal hypothesis that only a significance test over the right records can settle. The verifier must specify the null, select the test and decide which records bear on the hypothesis, and any of these choices can go wrong while the conclusion still comes out right. Yet benchmarks for language-model data analysis grade only the conclusion, with the relevant data handed over. We study the problem in health, where a wrong verdict is costly, and present STRATA, a benchmark that grades every decision on the way to a verdict: 8,735 hypotheses over the clinical timelines of vaccine adverse-event reports, with gold plans, evidence sets, p-values and verdicts derived under a released codebook and reproduced by a deterministic oracle. We also propose ART (articulate, retrieve, test), a staged paradigm that exposes each decision as an inspectable output. Extensive experiments with five model configurations show that systems fail mostly at choosing which records count, a verdict-evidence gap that accuracy alone conceals. ART closes most of this gap and brings a 31B open model to within 0.01 of the frontier models under the same paradigm; the frontier models succeed once given the codebook.
BibTeX
@misc{he2026testimony,
  author = {Chengyang He and Wenlong Zhang and Koduvayur Subbalakshmi and Ping Wang},
  title = {{Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses}},
  year = {2026},
  eprint = {2610.40619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40619},
}
Figure