Collapsing Calendar Time to Counter Recency Bias in Temporal Question Answering

Language-model agents that answer questions about dated events do well when the question names a day or a month, and noticeably worse when it names a year. The standard explanation is that year questions need more retrieved facts. We propose a sharper explanation and a fix that follows from it. Retrieved facts are usually listed in date order, so a fact's position in the prompt tracks its place in time. Language models tend to overweight whatever comes near the end of their input, which is called recency bias. When position and time line up this way, the model's pull toward late positions becomes a pull toward late events. The prediction is concrete: in year-scale questions, "who did X first" should fail more often than "who did X last", and reversing the order of the list should reverse that gap.

The fix borrows an old idea from temporal logic. At a coarse scale, a whole day can be treated as a single indivisible point, which Allen and Hayes call a "moment", and it can be opened back up into its parts when detail is needed. Our agent never reads a year's worth of daily facts in one pass. It first reads a short digest of twelve month-level moments, built deterministically from the knowledge graph. It then expands only the months it needs. Every sequence the model reads stays short enough that recency bias has little room to operate. The claim to test is that this deterministic coarse-to-fine collapse closes the year-level gap, and that it beats simpler ways of removing position effects at the same cost.

Core Claim

The year-granularity weakness reported for TempAgent comes largely from positional recency bias over long, chronologically ordered lists of retrieved facts. TempAgent is a ReAct loop, meaning the model alternates between reasoning steps and tool calls, and its nine Day/Month/Year tools filter by Before, After, or On. Its authors attribute the year weakness to aggregation volume. We claim it is mainly a position effect, and that granularity-matched context construction removes it more reliably than reordering, prompting, or shrinking the retrieval budget. The new element is the aliasing diagnosis combined with a symbolic magnification step. Neither appears in the multi-granularity question-answering work nor in the recency-bias work.

Why this is plausible, and what it builds on

The mechanism has theoretical support. LayerNorm Induces Recency Bias in Transformer Decoders proves that causal attention combined with LayerNorm (a per-token rescaling step) produces attention that grows as keys approach the query. That proof uses simplified parameter-free attention, so it motivates the hypothesis but does not establish it for a real agent. On Positional Bias of Faithfulness for Long-form Summarization reports a U-shaped faithfulness curve on long inputs and a "mode switch" toward the tail as context grows. That is the behavior we expect when a year of events is dumped into one prompt.

TempAgent's own ablation finds that about ten retrieved tuples is optimal. This is consistent with degradation beyond short lists, but year questions often need more than ten facts.

Three prior efforts connect the two directions without resolving this question. CoFi-Speech explicitly counters recency bias in speech generation with 120/40/20 ms coarse-to-fine token scales. It works on continuous learned acoustic tokens and never asks whether its coarse level is itself position-biased. MultiQA pools day-level facts into month and year embeddings with a Transformer, but never examines position effects in that pooling. Memoria fights recency-driven forgetting with tiered memories, but stores memories at a single, uniform granularity.

One negative result constrains our design. The summarization study found that hierarchical merging, where a model summarizes chunks and then summarizes the summaries, makes recency bias worse. Our digest differs on the property we think matters. It is computed symbolically from the graph: counts, the first and last event per month, and the relation types present. It is not written by the biased model. This follows the upward and downward conversions between granularities in An Algebra of Granular Temporal Relations, which preserve ordering relations across scales without needing numeric distances. Our reconciliation hypothesis is that hierarchy helps when the coarse layer preserves the ordering relations the question depends on, and hurts when the coarse layer is lossy and produced by the biased model. The experiments test this directly.

Research Program

The first part is a diagnostic. On MultiTQ Year-granularity questions, we hold the set of retrieved facts fixed and vary only their order: chronological, reverse-chronological, random, and relevance-sorted. We also move the gold supporting fact across positions to trace an accuracy-by-position curve for each granularity. We then summarize the result with one aliasing number. Let $A_{q}^{o}$ be Hits@1 (the fraction of questions whose top answer is correct) for question type $q$ ("first" or "last") under ordering $o$. The aliasing index is

$$\Delta = \left(A_{\text{last}}^{\text{chron}} - A_{\text{first}}^{\text{chron}}\right) - \left(A_{\text{last}}^{\text{rev}} - A_{\text{first}}^{\text{rev}}\right).$$

Pure positional recency predicts $\Delta > 0$, since reversal flips which question type gets the favored tail. Order-invariant reasoning predicts $\Delta \approx 0$. We compute the same curves at Day and Month granularity, where lists are short. If our account is right, the year gap should shrink when year lists are truncated to month-list length, even with oracle facts.

The second part is the method, which we call collapse-and-magnify. For a year-scoped question, a deterministic tool returns twelve month moments. The model picks the candidate months, and a second tool expands those months into day-level facts. If an expanded month still holds more than a fixed cap of facts, the same step recurses to weeks. The model's reasoning never sees more than a small number of items at once.

We also test whether recency bias reappears inside the twelve-item digest itself. This is the untested question CoFi-Speech leaves open about its coarse layer. We run the position sweep on digest entries.

Baselines

The naive reference point is the same backbone model given all retrieved year facts in chronological order, with a single prompt and no agent loop. This shows the raw size of the year gap and the raw value of $\Delta$.

The strongest existing method is TempAgent, run with its published tool set and prompts. Its reported result on MultiTQ is 70.2% Hits@1 with GPT-4, described as a 41.3% improvement over MultiQA. MultiQA itself reports 0.293 overall Hits@1 and 0.548 on Month-level Before/After questions. We found no reported per-granularity Year Hits@1 for TempAgent on Lacuna, so we will measure it ourselves. TempAgent's authors note that smaller open models struggle with its tool calling. We will therefore run TempAgent and our method on the same backbone: the largest open-weight model our budget allows, plus a GPT-4-class model if we have access. Both methods get the same retrieval index, the same cap on tool calls, and the same budget for tuning prompts on a held-out development split. Beating TempAgent means higher Year-level Hits@1 at no more model calls, with Day and Month accuracy not falling by more than noise, measured over three seeds and different prompt orderings.

Prog-TQA is a symbolic baseline. It generates programs in a knowledge-base query language with temporal operators such as FilterFirst. It reports 47.0% and 60.9% relative Hits@1 gains on single- and multi-constraint questions, but no raw scores were found on Lacuna. We use its released pipeline where available.

The ablation that isolates our key ingredient replaces the deterministic digest with a digest the backbone writes itself, month by month. Everything else stays the same. If the self-written digest matches the symbolic one, the reconciliation hypothesis fails. A second ablation removes the magnify step and answers from the digest alone, to check whether the gain comes from collapsing the context or from granularity-aware selection.

Why not just…?

Why not just shuffle the list, or average answers over several orderings? Averaging over orderings removes systematic position effects in expectation. It is the most direct competitor to our diagnosis. Its weakness is that no single ordering is short, so each pass still suffers the tail neglect the summarization study reports on long contexts. Averaging also multiplies cost. We will compare against five-ordering majority voting at a matched number of model calls. If voting matches collapse-and-magnify, we will report that the fix is debiasing, not granularity.

Why not just prompt the model to attend to early and middle facts? The summarization study found such focus prompts only "moderately effective," and they did not prevent the tail mode switch at long lengths. We include a focus-prompt condition in the diagnostic and check whether it drives $\Delta$ to zero. We expect it to shrink $\Delta$, not eliminate it.

Why not just hand the question to a symbolic program? For clean knowledge-graph questions like "first after X," a FilterFirst operator sidesteps position entirely. This is a real threat on MultiTQ. We will report per-question-type results against Prog-TQA and state plainly where it wins. The case for our approach rests on two limits of the symbolic route. Prog-TQA's authors report spurious programs that reach the right answer through wrong logic. And symbolic execution needs exact entity linking and explicit timestamps, whereas implicit markers like "thereafter" are exactly where ChronoFact reports failures. We therefore also evaluate on ChronoClaims, where evidence is text rather than query-ready tuples. ChronoFact reports 85.49 Macro F1 there, against 61.93 for zero-shot GPT-4o. We test whether month-level collapse of evidence events helps a language-model verifier.

Why not just retrieve fewer facts? TempAgent's $k=10$ optimum suggests shorter lists help. But a year question with forty relevant events cannot be answered from ten arbitrary ones. We compare accuracy against $k$ for flat retrieval and for our method. The prediction is that flat retrieval peaks and then declines, while ours keeps improving as more facts become available.

Decisive Checks

The account fails outright if $\Delta$ is near zero and the year gap persists when year lists are truncated to month-list length with oracle facts. That result would mean the weakness comes from aggregation reasoning, not position.

The method is supported if three things hold. Collapse-and-magnify must raise Year Hits@1 above TempAgent on the same backbone. It must beat ordering-averaged voting at matched cost. And the symbolic digest must clearly outperform the model-written digest. That last contrast is the one that would explain why hierarchy helped in CoFi-Speech but hurt in long-form summarization (On Positional Bias of Faithfulness for Long-form Summarization).