Memory That Blurs With Age: Log-Time Consolidation for Long-Horizon Agents

Conversational agents that remember users over months face a basic accounting problem. If they keep every past exchange word for word, storage and retrieval noise grow without limit. If they summarize on a fixed calendar (per session, per day, per week), they spend as much detail on a week from two years ago as on last week. We propose a memory that gets coarser the older it is. An event from an hour ago is stored nearly verbatim with its exact timestamp. An event from a year ago survives only as part of a consolidated fact whose time stamp is about as imprecise as a person's would be ("sometime last spring"). The width of each memory's time window grows in proportion to its age, which is the Weber law of human time perception. Total memory then grows with the logarithm of history length rather than linearly. A small, fixed budget of surprising or contradictory episodes is exempted and kept verbatim at any age.

The bet is that this schedule, borrowed from scale-invariant neural timelines, is a better consolidation policy for agent memory than calendar bins. It should match the accuracy of the best hierarchical memories at a fraction of their stored size. It should degrade gracefully as histories grow tenfold. It should also know when a temporal question cannot be answered from blurred memory: two events that have collapsed into the same coarse window can no longer be ordered, and the system should say so rather than guess.

Core Claim

Log-spaced temporal filter banks already work inside neural networks. Gradual Forgetting convolves token histories with 53 filters whose time constants grow geometrically. This lets a 124M GPT-2 attend over 8,192 tokens while computing attention over about 300 vectors, and it reaches 23.56 perplexity on WikiText-103 against 24.00 for Transformer-XL. That line of work, together with DeepSITH and the synaptic decay spectrum model, supplies our schedule. Those systems blur dense embeddings, though, and cannot store or query discrete facts. The authors of Gradual Forgetting state plainly that the compression "cannot retrieve verbatim quotes from the deep past."

Agent memories solve a different part of the problem. TiMem consolidates facts upward through a five-level tree (turns, sessions, days, weeks, persona). TSM clusters temporal knowledge-graph entities within monthly slices into topics and personas. RGMem coarse-grains when evidence counts cross a threshold. All three keep structured, queryable facts, but their consolidation boundaries are fixed calendar units or evidence counts, and none of them forget. TiMem's authors explicitly call fixed boundaries "a simplification" and point to adaptive scheduling as future work.

Our claim is that geometric time windows should be applied to the store of discrete facts itself. The window containing an event of age $a$ is indexed by

$$k(a) = \lfloor \log_\gamma (a/\tau_0) \rfloor ,$$

where $\tau_0$ is the finest resolution (for example, one minute) and $\gamma > 1$ is the growth ratio. Each window holds at most a fixed number of consolidated facts. Every fact carries an interval stamp as wide as its window. This yields three testable consequences. First, total stored memory grows as $O(\log T)$ in history length $T$, plus the pinned verbatim budget; this follows because the number of windows covering $[\tau_0, T]$ is $\log_\gamma(T/\tau_0)$. Second, the policy is approximately invariant to rescaling how often a user interacts, because dilating all ages by a factor $\alpha$ shifts every window index by the same constant, $\log_\gamma \alpha$. Calendar bins do not have this property. Third, temporal-order questions become explicitly decidable or undecidable depending on whether the two events' stored intervals overlap.

Research Program

The system has four parts. The first is a streaming extractor that turns each session into time-stamped facts of the form (subject, relation, object, time), separating when something was said from when it happened, as TSM does.

The second part is a log-structured consolidation step. As the clock advances, facts age into wider windows. When a window exceeds its budget, a language-model consolidator merges its facts into fewer, more general ones. Merged interval stamps take the union of their parts. This is temporal marginalization in the sense of Learning with Knowledge Graphs, applied gradually rather than all at once. We budget windows by fact count rather than by token length so that the stored-size comparison is clean.

The third part is surprise pinning. Following Nagy, Orbán and Wu, predictable experience should be compressed hard while surprising experience should be kept raw. We score each incoming fact by how poorly the current consolidated store predicts it, using the language model's log-probability of the fact given the retrieved semantic context. Facts that contradict the store receive extra weight, which also covers the abrupt preference changes RGMem's Tension term targets. The top-scoring facts, up to a fixed fraction of the budget, stay verbatim with exact timestamps regardless of age. This part addresses the tension between verbatim recall and lossy gist compression that neither compression line resolves on its own.

The fourth part is interval-aware answering. Temporal questions are resolved against interval stamps using Allen-style relations. The ontology comes from Allen and Hayes and the treatment of blurred "moments" from Short Time Periods, and we use them as the logic for when a coarsened event may be treated as a point. If two intervals are disjoint, "before" or "after" is answered directly. If they overlap only because of coarsening, the system looks for a pinned verbatim episode that settles the order. Failing that, it answers with explicit uncertainty.

Baselines

The naive reference points are full-history long context and flat dense retrieval over all past sessions. LongMemEval reports that long-context models lose 30% to 60% relative to an oracle given only the relevant sessions. We run both with the same reader model as every other system.

The strongest structured competitor is TiMem. It reports 76.88% accuracy on LongMemEval_S and 75.30% on LoCoMo, with a 52.20% reduction in recalled memory length. It is the closest design: hierarchical temporal consolidation that differs from ours mainly in using calendar boundaries and in never forgetting. On LoCoMo the highest number in the file is RGMem's 86.17%. TSM reports 74.80% on LongMemEval_S and 76.69% on LoCoMo. We will rerun TiMem and TSM with our reader and consolidator model, the same extraction prompts, and the same tuning budget over their thresholds. We will also report their published numbers alongside, since cross-model comparisons are otherwise unreliable.

Beating TiMem means one of two things. Either we match its accuracy within one point at no more than half its stored facts on LongMemEval_S, or we exceed it at equal stored size. On LongMemEval_M, where no reported result was found on Lacuna for any memory system, beating TiMem means a smaller accuracy drop from the S setting to the M setting.

Three ablations isolate the key ingredients. The first uses linear equal-width windows at the same total budget. The second uses TiMem-style calendar windows at the same total budget, with oldest-first forgetting added so that storage matches ours. The third keeps log windows but removes surprise pinning. A single-timescale exponential recency decay applied to retrieval scores serves as a fourth control.

Why not just…?

Why not just keep everything and use temporally filtered retrieval, since storage is cheap? This is the strongest threat. TempAgent-style filtering reaches 70.2% Hits@1 on MultiTQ with GPT-4 by checking time scope before reasoning (TempAgent). An uncapped store with good filters may simply win on accuracy. We treat the full-store TSM/TempAgent-style system as an upper reference and frame our claim as a frontier of accuracy against stored size. If log consolidation does not dominate the budget-matched calendar and linear variants on that frontier, the idea fails. If it matches full storage only at large budgets, it is a storage optimization rather than a reasoning gain, and we will report it as such. RGMem's evidence that uncompressed context can hurt accuracy suggests that pruning may help, but we test this rather than assume it.

Why not just add more levels to TiMem's calendar tree? A turn, session, day, week tree is already roughly geometric. The differences are that TiMem never discards fine-grained facts about old periods, and that its boundaries are tied to calendar units rather than to age relative to the question. The budget-matched calendar ablation and the rescaling test below separate these effects directly. If calendar windows plus forgetting match log windows, the Weber-law argument adds nothing and we will say so.

Why not just down-weight old memories with an exponential recency score? A single decay rate imposes one characteristic timescale. It either forgets last month too fast or keeps last year too sharply, whereas a geometric window spectrum keeps constant relative precision. The recency-decay control tests this on questions binned by the age of their evidence.

Proposed Experiments

We evaluate on LongMemEval_S and LongMemEval_M, which cover roughly 115k tokens and 50 sessions versus 1.5M tokens and 500 sessions, and on LoCoMo. We report overall accuracy, accuracy on the temporal-reasoning category, stored fact count, and recalled tokens per query. We also plot accuracy against stored size for every system as its budget varies.

For multi-granularity temporal reasoning, we replay ICEWS05-15 chronologically as an event stream and pose MULTITQ questions at their timestamps. Answers must come from each system's streamed memory rather than the full static graph. The static published numbers, MultiQA at 0.293 Hits@1 and TempAgent at 70.2%, serve as full-access ceilings rather than direct competitors.

The first decisive check is a time-dilation test. We stretch the gaps between LongMemEval sessions by factors of 0.25 to 4 while keeping the questions fixed. Log windows predict nearly flat accuracy across these factors. Calendar windows predict variation, because the same facts cross different day and week boundaries.

The second decisive check is calibration on order questions. Among "before/after" questions whose two events fall in overlapping stored intervals, we measure how often the system correctly abstains or recovers the order from a pinned episode, compared with baselines that silently guess.

The third decisive check is a growth curve. Stored size against history length should bend logarithmically for our system and grow roughly linearly for TiMem. Accuracy should fall more slowly than TiMem's from the S to the M setting.

We stop the project if the budget-matched calendar-with-forgetting ablation matches the log schedule on the first and third checks.