Language models answer questions about a changing world in three wrong ways that look the same on a leaderboard. They repeat an answer that used to be true (staleness). They repeat a loud, unverified claim they found in search results (rumor-following). Or they produce the right answer for the wrong reason, because it was in their training data or because they found an answer key online (leakage). Every existing fresh benchmark reports a single accuracy number, so these three failures can hide one another.
We propose a benchmark in which each question is asked at several past moments ("who leads X as of March?", "as of June?"). For each moment, the model may search only a frozen copy of social media posts written up to that moment. Because the correct answer changes from one moment to the next, a wrong answer can be traced to its source. If it matches an earlier moment, the model is stale. If it matches a later moment that the snapshot cannot yet support, the model is drawing on knowledge from outside the evidence. If it matches a claim that was later refuted, the model followed a rumor. Social media is the natural source here because events surface there first, and they surface in messy form.
Core Claim
Dating the evidence matters more than dating the question. Freezing a social media corpus at each query time and scoring which time slice a model's answer belongs to gives a contamination and obsolescence diagnostic that should survive paraphrasing and does not depend on a live, leak-prone web. The same design gives the first controlled measurement of how platform noise pushes retrieval-augmented models back onto their stored parameters. Retrieval-augmented generation (RAG) here means a model that pastes search results into its prompt before answering.
Why This Gap Is Real
Three lines of fresh-benchmark work motivate the proposal, and all three draw only on edited sources and score with a single answer per question. RealTime QA builds weekly questions from news quizzes. It found that 34% of errors come from retrieval failures, and that when retrieval comes up short, models fall back on outdated knowledge instead of abstaining. LatestEval uses GPT-4 to generate reading-comprehension items from the past week of BBC, arXiv and GitHub. LiveNewsBench tests search agents on post-cutoff news; accuracy falls from 74% to 21.5% when search is disabled. None of them uses social media. None asks about facts whose truth flips repeatedly. And because RealTime QA and LiveNewsBench rely on live web search, they inherit the problem shown in Search-Time Data Contamination: agents retrieved posted answer keys and scored 37.5% on contaminated Humanity's Last Exam items against 17.4% on clean ones.
Two papers constrain the design. Test of Time shows that a drop in accuracy after the training cutoff is a fragile contamination signal. Cloze questions decayed by 3.83 points for GPT-4o-mini, while rephrased questions built from the same arXiv sources showed no decay. Influence tracing found the source document as the most influential training example 77.5% of the time for cloze items but only 17.5% after rephrasing. Any probe based on accuracy level is therefore suspect, which is why we score which time slice an answer matches rather than how often the model is right.
SituatedQA shows that about 16.5% of information-seeking questions change answer over time. Models were 15 points less accurate on updated facts even when given updated Wikipedia text. Its date-indexed answers supply our seed questions. Mitigating Temporal Misalignment by Discarding Outdated Facts predicts how long a fact stays true and uses that prediction to discount confidence or trigger retrieval. It supplies our strongest staleness baseline.
The closest neighbor is TripleFact. It watches X and Instagram, waits for professional fact-checkers to issue verdicts, and swaps entity names to separate memorization from reasoning; it reports up to 23% inflation in conventional benchmarks. We differ in three ways. TripleFact classifies discrete claims as real or fake, whereas we evaluate open-ended answers about states that change. It uses social media only to choose topics, whereas we make the noisy posts themselves the retrieval environment. And it removes temporal signal from evaluation, whereas we turn time into the measuring instrument. TimeLMs shows that Twitter-trained models lose about 10% within a year. It is our evidence that social text is where drift first appears. It never audits whether its update stream leaked its own evaluation data, and our sealed snapshots are designed to rule that out.
Research Program
We first build volatile-fact chains. Each chain is an entity-relation pair with a dated sequence of true values. Chains are seeded from SituatedQA's temporally varying questions and extended with new chains drawn from Wikipedia revision history and fact-checker verdicts. The verdicts also give a set of claims that were later refuted. Following TripleFact's asynchronous validation, a chain enters the benchmark only after its most recent value has been independently resolved. We then gather time-stamped public microblog and forum posts that mention each chain's entity. These posts are left unfiltered on purpose: they contain the ads, bots, sarcasm and generic replies documented by Chen et al. and Ghosal et al.. For each chain we choose four to six query times, including at least one before the change and one in the window right after it, while the change is still visible only on social media. For each query time we index the snapshot containing only posts dated at or before that time.
Each answer receives one of five labels: as-of-correct, stale (matches an earlier true value), prescient (matches a value whose first supporting post comes after the query time), rumor (matches a later-refuted claim present in the snapshot), or abstain. The prescience rate, measured separately for chains that resolved before and after each model's training cutoff, is our contamination signal. Prescience on chains that resolved before the cutoff means the model's stored knowledge overrode the evidence. Prescience on chains that resolved after the cutoff should be near zero, so any residue points to leakage of the evaluation itself. This is a property of which time slice an answer belongs to, not of how hard the question is. We therefore expect rephrasing the question to leave it largely unchanged, and we test that expectation directly.
To isolate noise, each question is run under four versions of the corpus. The first is raw. The second is heuristically filtered using RedPajama's document quality signals (RedPajama). The third is filtered by a learned quality score in the style of the plausibility classifiers used for LCCC (Wang et al.). The fourth is a news-only snapshot matched to the same chains and dates.
Baselines
The naive reference point is closed-book answering with the query date stated in the prompt. Its answers should be stale before the cutoff and prescient for chains that resolved before the cutoff. It tells us how much of every other system's behavior is simply its stored knowledge.
The standard retrieval baseline is single-shot RAG over the snapshot: the top ten passages by lexical and dense retrieval, mirroring RealTime QA's top-10 setup. On RealTime QA, GPT-3 with Google Custom Search reported 66.5% multiple-choice accuracy and 34.6 exact match (EM). Those numbers come from a different benchmark and serve only to anchor expectations.
The strongest agentic baseline is a ReAct-style search agent with Search, Visit and Answer actions, as in LiveNewsBench, restricted to the snapshot. The best open-weight result reported there is DeepSeek V3.2 Thinking at 84.5% on LiveNewsBench.
The strongest staleness-specific method is the fact-duration router of Zhang and Choi. It reported 20.5 EM on SituatedQA while retrieving for only 45% of queries, and a 50 to 60% reduction in expected calibration error (ECE, the gap between stated confidence and actual accuracy). It is the most direct competitor on the staleness half of the problem.
There is no reported result found on Lacuna for any system on time-sliced social media snapshots. All numbers on the new benchmark will therefore be produced here.
Fairness conditions are as follows. Every system uses the same backbone models: one open-weight model in the 7 to 8B range, one larger open-weight reasoning model, and one commercial API. Every system gets the same snapshot index, the same per-question budget of tool calls and tokens, and the same prompt-tuning budget on a held-out development split of chains. The duration router is retrained on our chain dates.
Two ablations isolate the key ingredient, which is time-sliced scoring. In the first, the same items are scored against a single latest-value answer key, as existing benchmarks do. In the second, the same agents run on the live web instead of the sealed snapshot. The proposal counts as useful if two things hold. First, systems that score within a few points of each other under single-answer scoring should differ clearly in staleness, prescience or rumor rate. Second, the live-web condition should show prescience on chains that resolved after the cutoff that the sealed snapshot does not.
Why not just…?
Why not just use LiveNewsBench or RealTime QA with a newer cutoff? These benchmarks are fresh, but each question has one fixed answer and the search runs on the live web. The first means staleness and correct tracking cannot be told apart. The second leaves them exposed to the leakage that Search-Time Data Contamination documents. This alternative is a genuine threat on a narrower point: social media might add nothing beyond news. The news-only matched snapshot settles this. If staleness and rumor rates on social snapshots are indistinguishable from news snapshots, including in the window right after a change, then the social sourcing is unnecessary and we will report that.
Why not just filter the social corpus and treat noise as a preprocessing problem? This could work, which is why filtered snapshots are experimental arms rather than an afterthought. The literature on noisy dialogue data suggests filters fail in a particular way. The authors of Identifying Untrustworthy Samples and Learning to Converse with Noisy Data report that hard filtering discards valuable data, which is why they weight samples instead of deleting them. Here the risk is that filters delete exactly the informal early posts that announce a change. We measure the evidence that survives filtering in the window right after each change. If filtering cuts the rumor rate without raising the staleness rate, the noise problem is solved by preprocessing, and the benchmark's remaining value lies in its staleness and leakage axes.
Why not just compare accuracy before and after the training cutoff? Test of Time shows that rephrasing erases this signal while the underlying memorization persists. We run the comparison anyway as a reference, and we test our alternative directly by paraphrasing every question three ways with different models and checking whether prescience rates move. If they move as much as accuracy-based decay does, our claim of robustness to paraphrasing fails.
Why not just add a fact-duration model and let the system abstain? Duration routing addresses staleness, but its duration estimates come from Wikipedia and it has no notion of how reliable the evidence is. It is run as a baseline. Our prediction is that it lowers the staleness rate but leaves the rumor and prescience rates unchanged, because those failures come from the content of the evidence rather than its age.
Decisive Checks
The central experiment uses roughly 1,500 chains with about five query times each, run through all systems and all four corpus versions. Three results would confirm the proposal. First, rumor and stale rates should rise from the filtered snapshots to the raw ones. That would be the first measurement of whether noisy evidence pushes models back onto stored knowledge, the parametric fallback that RealTime QA observed only in clean news settings. Second, for chains that resolved after the cutoff, prescience should stay near zero in the sealed snapshot but become nonzero on the live web, which would measure leakage directly. Third, prescience rates should hold steady under paraphrase where accuracy decay does not.
As a smaller verification of the leakage signal, influence tracing on OLMo2-7B-Instruct, using the EK-FAC method from Test of Time on about 40 chains, would check whether prescient answers trace to pretraining documents dated after the query time. A null result on any of these three predictions would identify which part of the design does not hold up.
References
- RealTime QA: What's the Answer Right Now?
- LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction
- LiveNewsBench: Evaluating Web Search Agents with Freshly Curated News
- Search-Time Data Contamination
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- TripleFact: Defending Data Contamination in the Evaluation of LLM-driven Fake News Detection
- TimeLMs: Diachronic Language Models from Twitter
- Microsoft Research Asia at the NTCIR-13 STC-2 Task
- Aesthetic Image Captioning From Weakly-Labelled Photographs
- RedPajama: an Open Dataset for Training Large Language Models
- A Large-Scale Chinese Short-Text Conversation Dataset
- Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization
- Learning to Converse with Noisy Data: Generation with Calibration