Language models that answer questions from a long history (chat logs, agent trajectories, retrieved documents) get worse when they are given too much context. Accuracy rises as relevant material is added, peaks, and then falls as distracting text accumulates. Current fixes treat the retrieved material as a flat ranked list and learn how many items to keep. We propose a different explanation and fix. The harm comes from off-topic text, not from volume. If that is right, the useful unit of retrieval is a topically pure cluster of memories, and the number of clusters should not be a hand-picked constant. It should be set by how much same-topic text the downstream model can absorb before its accuracy drops.
Concretely, we build a streaming memory that assigns each incoming dialogue turn to exactly one topic cluster. It opens new clusters as new topics appear and splits any cluster that grows past a token budget measured from the answering model itself. At question time it hands the model one cluster, occasionally two. This joins three lines of work that have stayed separate. The first measures the "inverted U" of context volume. The second notes that fixing the number of topics $K$ in advance breaks under drift. The third shows that forcing each short text into a single topic regularizes sparse data. Our contribution is a measured link between them: the reader's tolerance for distraction determines the right topic granularity.
Core Claim
The claim has two parts. The first is empirical. At a fixed context length, a reader model's accuracy depends mainly on the share of tokens that belong to the question's topic, so the peak of the inverted U moves to larger volumes as purity rises. The second part is constructive. If the first holds, a single-topic streaming clustering whose cluster sizes are capped at the reader's measured same-topic budget should match or beat learned volume selectors on long-term memory QA, without a model call at ingestion time.
The first part is the decisive scientific question, and the file contains no direct test of it. Beyond RAG vs. Long-Context introduces LDAR, which learns a band of similarity ranks to feed the reader and shows that readers differ in how much distraction they tolerate. LDAR measures that tolerance over an unstructured rank list. BudgetMem finds that 5 retrieved chunks beat 9. The temporal knowledge-graph QA systems TimeR⁴ and Time-aware ReAct find that about 10 facts is optimal. All three motivate the proposal: each finds a threshold but expresses it as a count, never separating count from topical composition. Older summarization work hints at our version of the claim. Single Document Summarization with Document Expansion found that adding topically related neighbors helps until unrelated topical material creeps in.
Research Program
The memory model is a streaming single-topic mixture. Each dialogue round, meaning one user message plus one assistant reply (the granularity LongMemEval found works best for storage), is embedded with a sentence encoder. The round is then placed into one cluster by a greedy rule from Dirichlet-process mixtures, a family of clustering models that can create new clusters as data arrives. Round $x_t$ joins the existing cluster $k$ that maximizes $\log n_k + \log p(x_t \mid k)$, where $n_k$ is the cluster's current size and $p(x_t \mid k)$ is a von Mises–Fisher likelihood around the cluster's mean direction (a Gaussian-like density for unit vectors). The round instead opens a new cluster when $\log \alpha + \log p_0(x_t)$ is larger, where $\alpha$ controls how readily new topics are created and $p_0$ is a broad prior.
The departure from standard nonparametric clustering is a budget constraint. No cluster may exceed $B^*$ tokens. A cluster that crosses the limit is split by two-means clustering on its members. The budget $B^*$ is measured for each reader model:
$$B^* = \arg\max_{B}\; \mathrm{Acc}_{\text{reader}}(B \mid \text{purity}=1),$$
that is, the context length at which the reader's accuracy peaks when every token comes from the question's topic. We estimate it on a held-out calibration split. If our empirical claim fails and purity does not matter, $B^*$ reduces to an ordinary count threshold, and the method should then do no better than tuned top-$k$. The design makes this falsifiable.
At query time the question is assigned to its single most probable cluster, and that cluster is passed to the reader in chronological order. This responds to the LDAR observation that ordering is unaddressed. The direction overview identifies a weakness of single-topic assignment: it cannot represent texts that span two concepts. Multi-session questions are exactly that case. To handle them, a second cluster is added when the second-best posterior is within a margin $\delta$ of the best, and only if the combined size stays under $B^*$.
This design takes from the short-text literature without copying it. Dynamic Clustering of Streaming Short Documents (DCT) supplies the evidence that one-topic-per-document assignment stays stable over a 369M-tweet stream. Its conclusion names fixed $Z$ as the limitation, and we remove that limitation. CME-DMM shows the single-topic constraint beating LDA on short texts (Micro-F1 0.87–0.88 versus 0.57). Like DCT, it works on word counts with fixed $K$ and never feeds a generator. We move the mixture onto embeddings because dialogue rounds share little vocabulary. K-means for Evolving Data Streams constrains the design: fixed-$K$ streaming models distort old clusters to absorb new concepts, which is why our split and birth operations must leave existing assignments untouched. Finally, The Sensitivity of Topic Coherence Evaluation to Topic Cardinality shows that human coherence judgments are diluted as more topic words are included, and that aggregating over several cardinalities makes automated metrics more reliable. We follow that lesson by reporting cluster quality across several budget values rather than one.
Baselines
The naive reference point is the reader given the full history. On LongMemEval_S this is the setting that LongMemEval reports as falling 30% to 60% below an oracle that sees only the evidence. We run it with the same reader and prompt as every other system and also report the oracle, which gives the ceiling.
The second reference is fixed top-$k$ dense retrieval over rounds using the same encoder. $k$ is tuned on the calibration split with the same number of trials used to estimate $B^*$, so that both methods get an equal tuning budget. This is the method BudgetMem showed peaks near 5 chunks.
The strongest existing published number in the file is BudgetMem's capacity-tiered variant. On LongMemEval it reaches a Judge Score of 60.50, against 48.51 for LightMem, as reported in BudgetMem. We take it as the system to beat on LongMemEval. Because the judge model and reader must match for scores to be comparable, we will rerun BudgetMem with our reader and judge instead of borrowing its number. Beating it means a higher Judge Score under identical reader, judge, and evaluation script, with no more online tokens.
LightMem is run from its public code, zjunlp/LightMem. It is the closest structural competitor because it also groups dialogue by topic. Its groups, however, are contiguous segments cut at semantic boundaries, whereas ours collect rounds from distant sessions that return to the same topic. It reports accuracy gains of up to 29.29% on LoCoMo.
LDAR is the strongest competitor on the scientific question, since it learns the reader's distraction tolerance directly from score distributions. The file reports no LongMemEval or LoCoMo number for LDAR, so we train it on our calibration split with the same reader and compare there.
Two ablations isolate the key ingredient. The first keeps our clusters but sets the budget by count, using the tuned top-$k$ value rather than the purity-1 peak. The second replaces the single-topic assignment with soft membership, where each round contributes to its top three clusters. If neither ablation loses ground, the budget principle and the single-topic constraint add nothing beyond tuning.
Why not just…?
Why not just tune $k$ per reader and stop? A global $k$ treats every question alike, but the amount of same-topic evidence varies widely. A preference question may need one round, while a knowledge-update question may need every round on that topic across months. Our experiments compare global tuned $k$, per-question oracle $k$, and cluster retrieval. If the cluster method does not close a meaningful part of the gap between global $k$ and oracle $k$, the simpler answer wins and we will say so.
Why not just use LDAR's learned rank band? This is the most serious threat. LDAR adapts volume per query using only similarity scores, and it might already capture purity indirectly, since a sharp drop in scores signals a topic boundary. The test is direct: we train LDAR and our method on the same calibration data and then evaluate on questions grouped by how scattered their evidence is across sessions. Cluster retrieval should help most when evidence is spread far apart in time but shares a topic, which is where a similarity band picks up near-duplicate distractors. AssoMem found that pure semantic similarity fails in such similarity-dense memories. If LDAR matches us on that slice, our structural claim is not supported.
Why not just run an off-the-shelf clustering such as agglomerative clustering with a distance cutoff? This may well work as well as our model, and we do not argue around it. We include agglomerative clustering on the same embeddings, with its cutoff chosen so that median cluster size equals $B^*$. If it matches the probabilistic model, the contribution is the budget-matched cardinality principle, not the mixture likelihood. That would still be a useful finding, and it would simplify the system.
Why not just ask a model to label each turn's theme, as STITCH does? Grounding Agent Memory in Contextual Intent does exactly that and reports a 35.6% gain over its strongest baseline on CAME-Bench. Its authors note, however, that it needs several model calls per ingestion step and struggles to trade off schema stability against adaptation. We report ingestion calls and tokens alongside accuracy. Our goal is comparable accuracy at zero ingestion calls, not to dominate STITCH on accuracy.
Proposed Experiments
The first experiment tests the empirical claim. On LongMemEval_S we build contexts of fixed token length ranging from 2k to 64k. For each length we vary topical purity, meaning the fraction of tokens from sessions on the question's topic, determined from evidence annotations plus our clusters. We then measure accuracy for one small open-source reader and one larger reader. The prediction is that the location of each reader's accuracy peak grows with purity. A flat response to purity at fixed length would falsify the core claim, and we would stop before building the constructive method.
The second experiment is the main comparison. It runs on LongMemEval_S, LongMemEval_M, and LoCoMo. All baselines above share the same reader, judge, and token accounting. We report Judge Score, online tokens, and ingestion calls, with question-type breakdowns for multi-session and knowledge-update questions.
The third experiment tests drift and the fixed-cardinality question. LongMemEval_M's roughly 500 sessions are replayed in order. We compare our model against the same model frozen at the $K$ it had reached after the first 50 sessions, which mimics DCT-style fixed $Z$. We track accuracy on questions whose evidence comes from topics first appearing late in the stream. The prediction is that frozen $K$ degrades specifically on those questions, reproducing in agent memory the distortion documented for streaming k-means.
The decisive outcome is the combination of three results: the purity-shifted peak, a Judge Score above a rerun BudgetMem at equal tokens, and an advantage over LDAR concentrated on topically scattered evidence. Together these would show that topic cardinality is best set by the reader's noise threshold. Any one of them failing tells us which part of the story is wrong.
References
- Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
- Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory
- TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering
- Time-aware ReAct Agent for Temporal Knowledge Graph Question Answering
- Single Document Summarization with Document Expansion
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
- Dynamic Clustering of Streaming Short Documents
- Collaboratively Modeling and Embedding of Latent Topics for Short Texts
- K-means for Evolving Data Streams
- The Sensitivity of Topic Coherence Evaluation to Topic Cardinality
- LightMem: Lightweight and Efficient Memory-Augmented Generation
- AssoMem: Scalable Memory QA with Multi-Signal Associative Retrieval
- Grounding Agent Memory in Contextual Intent