Belief-Weighted Agent Memory: Tracking Uncertain User Intent to Keep Distracting History Out of the Prompt

Language-model agents that work with a user over long sessions keep a memory of past turns and pull pieces of it back into the prompt when needed. The pieces they pull back are often wrong. The same hotel, price, or person comes up under different goals, so a search for "similar text" returns turns that look relevant but belong to another task. Language models are known to be easily thrown off by irrelevant text in their prompt. They build false links between unrelated passages and the question, and reasoning models can even use their extra thinking to justify misleading context. The best current fix labels each turn with the user's current goal and retrieves only memories that carry a matching label. That fix fails whenever the label is wrong, and on ambiguous turns the label is often wrong.

We propose to treat the user's current goal as a hidden variable, in the way user-modeling research has long treated clicks and pauses: as noisy signals of an intent that cannot be observed directly. The agent keeps a probability distribution over possible goals, which we call a belief. It updates that belief turn by turn with a small hidden Markov model, meaning a model in which a hidden state changes over time and each observation is a noisy readout of it. It also revises the goal labels on stored memories once later turns resolve earlier ambiguity. At retrieval time, a memory is scored by how much its goal distribution overlaps the current belief, and the number of memories admitted into the prompt grows with how uncertain the belief is. The claim is specific and testable. Probabilistic intent tracking with a temporal prior should beat deterministic intent labels on long, interleaved agent trajectories. The margin should widen as turns become more ambiguous, and it should come at lower per-step cost than extracting labels with a large model.

Core Claim and Relation to Prior Work

The closest existing work is STITCH, described in Grounding Agent Memory in Contextual Intent. STITCH tags every step with a three-part intent label (the overall scope such as "Day 2 itinerary", the event type such as "hotel booking", and the key entity types). It then filters memory candidates whose labels are incompatible with the current step, and reports beating its strongest baseline by 35.6% on the largest subset of CAME-Bench. STITCH motivates this proposal because it shows that intent, not surface similarity, is the right key for filtering memory. It also exposes the gap we target. Its labels are deterministic, a mislabeled scope silently suppresses valid history, and it needs several large-model calls per step to produce the labels. A second close work is DFA-RAG, which builds a fixed finite-state automaton of dialogue stages offline and retrieves examples tied to the active stage. It reports a 90.0% success rate on MultiWOZ. When an input does not fit the automaton, DFA-RAG falls back to the last valid state, so it has no way to express that the user might be in one of two stages. We keep the stage-conditioned retrieval idea from both papers. What we add is uncertainty over the state, a learned temporal prior, and retroactive relabeling. Neither paper does any of the three.

The latent-state side supplies the method. User Model and Utility Based Power Management uses a dynamic Bayesian network to tell a thinking pause from real idleness instead of applying a fixed threshold. Exploring searcher interactions for distinguishing types of commercial intent infers hidden research-versus-purchase states from cursor and scroll behavior. Expectation-Maximization Method for EEG-Based Continuous Cursor Control fits latent micro-intentions with EM under very noisy signals. The shared lesson is that a temporal prior over a hidden state smooths out individual misleading observations, and a deterministic per-step classifier cannot do that. We reuse this structure: sticky transitions, noisy emissions, and EM fitting. The recommendation POMDP papers (Optimal Recommendation to Users that React, User Study of Spoken Decision Support System) show belief tracking used to choose actions. Misu et al. also constrain our design: their dialogue system resets its belief between sessions and degrades on returning users. For that reason we attach beliefs to stored memories rather than only to the live session.

The noise-sensitivity literature sets the stakes and rules some designs out. Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents shows that removing distracting documents raises exact match by an average of 64% over naive zero-shot readers. Thinking in a Crowd shows Qwen3-8B falling to 77.03% with two misleading snippets, against a 96.22% baseline. This is why "admit more memory and let a reasoning model sort it out" is not a safe fallback. Self-Knowledge Guided Retrieval Augmentation finds that simply asking the model whether it needs retrieval is the weakest strategy, which argues against using the agent's own verbal confidence as the uncertainty signal. We therefore calibrate a separate small model instead.

Research Program

The hidden state $s_t$ at turn $t$ is a pair: a thematic scope and an event type, using STITCH's categories so the comparison is direct. The number of scopes is open-ended, so the transition model has a "new scope" state. A newly opened scope becomes a fresh state, in the style of a sticky hidden Markov model that tends to stay in its current state. Transitions are learned with EM from unlabeled trajectories, starting from a high self-transition probability. Emissions come from a small encoder (a few hundred million parameters). It is distilled from large-model intent labels and outputs a calibrated distribution over states given the turn text and the previous turn. We convert that output to a likelihood by dividing by the class prior. The forward pass gives the belief $b_t(s) = P(s_t = s \mid x_{1:t})$, where $x_{1:t}$ are the turns so far. Each stored memory $m$ keeps a smoothed posterior $q_m(s)$. This posterior is recomputed with the forward-backward algorithm over a short trailing window, so a turn that looked ambiguous when written gets relabeled once later turns clarify it. Retrieval ranks memories by

$$\text{score}(m, t) = \text{sim}(m, x_t) \cdot \Big(\epsilon + \sum_s b_t(s)\, q_m(s)\Big),$$

where $\text{sim}$ is ordinary embedding similarity and $\epsilon$ is a small floor that keeps any memory from being ruled out entirely. The number of memories admitted is $k_t = k_0 + \lceil \alpha H(b_t) \rceil$, where $H$ is the entropy of the belief and $k_0, \alpha$ are tuned on a development split. A confident agent therefore sees a tight context, and an uncertain one hedges across hypotheses.

The analysis has three parts. The first compares retrieval precision and recall against gold relevant memories. The second is a stress test in which we corrupt the emission model's inputs, or swap in weaker taggers, to trace how answer accuracy degrades as labels get noisier. The third measures the cost per step in large-model calls and tokens.

Baselines

The naive reference point is plain dense retrieval over the full memory with a fixed top-$k$. We also run a full-history condition that places the whole trajectory in a long-context model's prompt. Both use the same embedding model and answering model as every other system. We expect our method to beat both by a wide margin, and a result that did not would mean the benchmark is not measuring distraction at all.

The strongest existing method is STITCH (Grounding Agent Memory in Contextual Intent). The file reports its relative gain of 35.6% over the strongest baseline on CAME-Bench's largest subset. It reports no absolute accuracy, so no reported result was found on Lacuna for STITCH's absolute CAME-Bench score, and we will rerun it ourselves. For LongMemEval and LoCoMo the file gives no numbers, so again no reported result was found on Lacuna for those. Fairness comes from sharing everything except the intent mechanism: the same answering model, embedding model, memory store, coreference step, and retrieval budget in tokens. To beat STITCH, we need higher answer accuracy on CAME-Bench's long-trajectory subsets at equal or smaller retrieved-token budgets. Separately, we should match its accuracy using at least three times fewer large-model calls per step. We also include DFA-RAG (DFA-RAG) on MultiWOZ, where it reports 90.0% success. That setting has little ambiguity, so there we aim only to match it, which checks that our method does no harm on clean workflows. A cross-encoder reranker on top of dense retrieval, in the style of Coarse-to-Fine Knowledge Selection, is a strong intent-free baseline. We give it the same development-set tuning budget.

Two ablations isolate the key ingredient. The first, "soft labels without time", keeps the calibrated per-turn distribution but drops the transition model and the smoothing. The second, "belief without relabeling", uses forward filtering only, so stored memories keep the labels they had when written. A third ablation fixes $k_t$ at its average value to test whether the entropy-adaptive budget matters.

Why not just…?

Why not just give STITCH top-2 labels or a confidence threshold? This is the most serious threat, and the "soft labels without time" ablation is built to test it. If per-turn soft labels recover most of the gain, then the contribution shrinks to "calibrate your tagger", and we would report it as such. Our prediction that the temporal prior matters rests on the structure of CAME-Bench's interleaved, non-turn-taking trajectories. There, a single turn often cannot identify the thread it belongs to, but its neighbors can. That is the situation where the dynamic Bayesian network in User Model and Utility Based Power Management beats fixed thresholds. The gap between our full method and this ablation should widen with trajectory length and with how often threads interleave. If it does not, the claim fails.

Why not just use a strong reranker conditioned on the last few turns? A cross-encoder that reads recent context does capture some intent implicitly, and the Re3G reranker contributes about 3.5 F1 on Multidoc2dial (Coarse-to-Fine Knowledge Selection). It has no memory of which thread is active beyond its input window, however, and it cannot relabel old memories. STITCH's large gain over similarity-based baselines suggests that the missing variable is explicit goal state. We run the reranker as a baseline and also stack it under our score, which shows whether the two are complementary.

Why not just put everything in a long context and let a reasoning model filter it? Long context windows exist, but Thinking in a Crowd shows that misleading context can pull a thinking model from 96.22% down to 77.03%. Improving Zero-shot Reader by Reducing Distractions shows large gains simply from removing distractors. The full-history condition measures this directly on our benchmarks, with thinking both on and off.

Why not just ask the agent how confident it is about the user's goal? Self-Knowledge Guided Retrieval Augmentation found direct self-assessment to be the weakest way to decide about retrieval. We still include a variant in which the large model states a goal distribution verbally, replacing the distilled emission model. This tests whether calibration from a separate model is actually needed.

Proposed Experiments

The main evaluation runs on CAME-Bench, with results reported by trajectory-length bucket, because STITCH's gains grow with length and ours should too. We add LongMemEval and LoCoMo for breadth and MultiWOZ as the low-ambiguity check against DFA-RAG. We report answer accuracy, retrieval precision and recall against gold supporting turns, retrieved tokens per query, and large-model calls per step. Each result is averaged over three seeds with bootstrap confidence intervals.

The first decisive check is the label-noise curve. We degrade the intent tagger in controlled steps and plot answer accuracy. STITCH's curve should fall steeply, and ours should fall slowly. If the curves are parallel, the belief machinery adds nothing beyond a better tagger. The second is the relabeling test, run on the subset of turns whose smoothed label differs from the label assigned when they were written. On those turns, retrieving them later should succeed more often under our method than under forward filtering alone. The third check is whether the entropy-adaptive budget places more tokens on turns where the answer truly depends on several threads, and fewer elsewhere, while keeping accuracy equal to a fixed budget at lower average cost. Taken together, these results would say whether probabilistic intent tracking with a temporal prior is the thing that keeps distracting history out of the prompt, or whether calibrated per-turn labels already do the work.