Where Did the Truth Change? Relative-Entropy Change Points as Explanations for Time-Sensitive Claims

Many false-looking claims are really outdated ones. "X is the prime minister" or "drug Y is recommended for Z" was true when first said and later stopped being true. Fact-checkers then need to answer two questions: is this claim supported, and if not, did it ever hold and when did that end? Current automated systems answer neither well. They either ignore time or reduce it to a date feature, and their explanations are either free-text rationales or extracted sentences. None of these can point to the moment the evidence changed.

This note proposes a way to find that moment. We take all dated evidence mentioning the claim's entities and slice it into time windows. We then measure how surprising each window's word distribution is given the window before it, using Kullback-Leibler (KL) divergence, which is the extra cost of encoding one distribution with a code built for another. Peaks in this surprise mark candidate change points. Because KL divergence splits exactly into per-word contributions, each peak also comes with the words that caused it, such as "resigned" or a successor's name. The explanation handed to the verifier and the human reader therefore has three parts: a date, the vocabulary that drove the shift, and the sentences that contain that vocabulary. Every part is computed directly from the evidence, not generated.

Core Claim

We claim that claim-conditioned, windowed relative entropy over dated evidence gives a change-point signal that does three things. It improves veracity prediction on outdated claims. It flags evidence carrying suspicious timestamps. Its explanations are more faithful than LLM-written temporal rationales, because each element can be traced back to evidence counts.

What is new is the combination of three things no prior work combines: diachronic change detection, per-claim evidence streams, and use of the per-feature decomposition as a user-facing explanation.

Grounding in Prior Work

The method comes from diachronic linguistics. Using relative entropy for detection and analysis of periods of diachronic linguistic change computed KL divergence between adjacent sliding windows of the Royal Society Corpus. It showed that peaks mark rapid change and that per-feature contributions isolate the specific innovations, such as "oxygen" replacing "oxygene." That work ran over centuries of curated text and a whole-corpus vocabulary.

We move the same machinery to a much smaller and noisier object: the stream of web documents about one claim's entities, over months rather than decades. This change makes sparsity the central problem. Two lines of work establish that the Jensen-Shannon (JS) divergence, a symmetric KL variant measured against the average of the two distributions, stays stable on sparse linguistic distributions: Feature Weighting for Co-occurrence-based Classification of Words and Clustering Polysemic Subcategorization Frame Distributions Semantically. We borrow JS as the smoothed fallback. Exploring variation across biomedical subdomains motivates computing divergence per claim rather than globally, since it showed that one broad "domain" hides sharply diverging sub-distributions.

Temporal fact-checking motivates the problem and supplies the strongest competitors. Implicit Temporal Reasoning for Evidence-Based Fact-Checking added bucketed claim-to-evidence date distances to MultiFC classifiers. This showed that time matters a great deal (a BiLSTM rose from 32.39% to 47.43% Macro F1), but distance is a proxy: it treats a document from 400 days later the same whether or not anything changed in between. Our signal instead asks whether the evidence itself shifted.

TACV, from Evidence-Based Temporal Fact Verification, and ChronoFact: Timeline-based Temporal Fact Verification extract events and dates and check their chronological order. ChronoFact enforces the ordering with soft logic. Both assume explicit event-time tuples can be extracted, and both report trouble with implicit temporal references. Our method never parses dates inside the text. It uses only document timestamps and word distributions, so it complements them rather than replacing them.

HALO: Half Life-Based Outdated Fact Filtering in Temporal Knowledge Graphs filters outdated facts with a fixed parametric decay. We replace the assumed decay with an observed change point.

Explainable fact-checking constrains the design of the explanation. Generating Fact Checking Briefs found that about 45% of crowdworkers accepted briefs without searching further. That is a strong reason to prefer explanations whose parts can be checked against sources. e-FEVER: Explanations and Summaries forAutomated Fact Checking showed that abstractive summaries carry useful signal but can echo the claim when evidence is missing. LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification showed that decomposed, faithful verdicts can localize a "culprit" phrase. We aim for a temporal analogue of that: a culprit window plus culprit terms.

AVeriTeC supplies the setting and a known failure mode. AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web splits its data chronologically and warns that search APIs return wrong "published on" timestamps, which lets future evidence leak into the past. This leakage is a direct use case for our surprise signal.

Research Program

For a claim $c$ with timestamp $t_c$, we retrieve dated evidence and keep only sentences mentioning the claim's entities, resolved by an entity linker. We build feature distributions $P_w$ over sliding windows $w$. Features are lemmas and entity mentions, with a frequency cutoff, following the diachronic work. For consecutive windows we compute each feature's contribution to the divergence:

$$ D_{KL}(P_{w+1} \parallel \tilde P_{w}) = \sum_{f} P_{w+1}(f)\,\log \frac{P_{w+1}(f)}{\tilde P_{w}(f)}, $$

Here $f$ ranges over features, and $\tilde P_w$ is the earlier window's distribution interpolated with a background news distribution so that no feature gets zero probability. We report JS divergence as a check on robustness. The summed terms give a divergence curve over time. The largest individual terms at a peak are that change point's "driving features."

To separate real changes from ordinary news bursts, we calibrate each peak against a null distribution. The null comes from the same claim's stream with timestamps permuted within a local neighborhood. This gives a per-peak significance score, which is also how we set thresholds instead of choosing them by hand.

Consumer 1: the verifier. The change-point report feeds a verifier. We test two versions: a RoBERTa classifier and a fine-tuned open LLM in the HerO style. The report contains the significant change points relative to $t_c$, their driving features, and the evidence sentences with the highest contribution mass. We extend the label space where the data allows, separating "supported at $t_c$ but superseded" from "refuted."

Consumer 2: a leakage detector. A document dated before $t_c$ is flagged if its features carry high surprise under the pre-$t_c$ windows but low surprise under a post-change window. Such a document is likely misdated.

Consumer 3: the explanation. The explanation shown to users is a template: a date, a ranked list of terms, and the linked sentences. It involves no free generation. An optional LLM paraphrase may be layered on top, but it is evaluated separately.

Baselines

The naive reference point is a verifier that sees evidence with no temporal information. On MultiFC, MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims reports 44.1% Macro F1 for evidence-based models without metadata and 49.2% for its best model with metadata.

The strongest temporal competitor on MultiFC is the date-distance grounding of Implicit Temporal Reasoning for Evidence-Based Fact-Checking, reported at 47.43% Macro F1 with a BiLSTM. Its exact RoBERTa number is not in our file. We will reimplement it on the same encoder, splits, and hyperparameter budget as our model. Beating it means a statistically significant Macro F1 gain on the full test set and a larger gain on the subset of claims whose evidence spans a detected change point.

On AVeriTeC, the strongest open system is HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims. It reports an overall shared-task score of 0.57, an answer score of 0.2801, and 75.2% development-set veracity accuracy. For reference, AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web reports 0.39 veracity F1 for GPT-3.5-turbo. We will rerun HerO's verifier with and without our change-point report, holding retrieval, model size, and fine-tuning steps fixed. Beating it means higher accuracy on development-set claims that have post-claim evidence and a lower rate of verdicts justified by leaked evidence.

For change-point localization, no reported result was found on Lacuna. The comparison points are a document-volume burst detector and a detector based on raw frequency differences.

The key ablation replaces the KL contribution with an unweighted frequency difference, and separately replaces the calibrated threshold with a fixed one. This isolates whether relative entropy itself matters, rather than simply having any change signal.

Why not just…?

Why not just filter evidence by date? A hard cutoff at $t_c$ answers "was it true then," and it is the right default. It fails in two ways. First, it trusts timestamps, which AVeriTeC shows are often wrong; our leakage detector targets exactly this case. Second, it discards the post-claim evidence a fact-checker needs to say that something is now outdated. We include cutoff filtering as a baseline. If cutoff filtering combined with a static verifier matches us on outdated claims, the verdict-accuracy part of our claim fails and only the explanation contribution remains.

Why not just ask an LLM to compare old and new evidence? This is a real threat for accuracy. The zero-shot numbers in the file are weak (GPT-4o reaches 61.93 on ChronoClaims against ChronoFact's 85.49), but a fine-tuned HerO-style model with dated evidence may still do well. Our claim here rests on faithfulness rather than accuracy. We test it by deleting the evidence sentences each explanation cites and measuring how the verdict changes, and by asking annotators to check whether the stated change date matches the sources. We expect LLM rationales to state plausible but unsupported dates more often.

Why not just detect spikes in document volume? Volume spikes catch news cycles, but a fact can change quietly, and a burst of coverage can happen with no change at all. KL divergence responds to what is said, not how much. We test this directly with the volume baseline on the localization probe described below.

Why not just use date-distance buckets? Distance cannot tell a stale-but-unchanged fact from a changed one. We evaluate this on the stratified subsets defined in the experiments.

Proposed Experiments

Change-point localization. Because no benchmark exists, we build a probe set from Wikidata facts with start and end qualifiers, such as officeholders, team memberships, and CEOs. We pair each fact with timestamped news mentioning its entities. The target is the qualifier's end date. The metric is the fraction of detected peaks within a tolerance window of that date, compared against the volume and frequency-difference detectors. This set is new, and we will release it.

Downstream verification. We run this on MultiFC and on the chronological AVeriTeC splits. Results are stratified by whether the claim's evidence spans a significant change point. The decisive check is that gains concentrate in the spanning stratum. If gains are uniform, the signal is acting as generic regularization rather than temporal reasoning.

Leakage detection. We inject evidence documents with shifted timestamps, and additionally audit AVeriTeC evidence whose dates can be manually verified. We measure precision and recall of our flags against a baseline that uses only the date field.

Explanation quality. We run a human study modeled on Generating Fact Checking Briefs. Crowdworkers verify time-sensitive claims using one of three aids: the change-point explanation, a HerO-style chain-of-thought rationale, or no aid. We measure accuracy and time. We also measure the rate of accepting the aid when it is wrong, which we create by occasionally corrupting the aid. The hypothesis is that checkable dates and term lists lower blind acceptance below the 45% seen for briefs.

The project fails if calibrated KL peaks do not beat volume spikes at localization. Every downstream claim depends on that result, so it is the experiment we run first.