Event-Driven Forgetting for Personalized Conversational Search

Conversational search assistants now receive a written profile of the user, a short list of facts such as "I am vegetarian," "I live in Edinburgh," or "I'm planning a trip to Japan," and use it to rewrite each question before searching. The profile is treated as fixed for the whole conversation. People do not behave that way. Some facts last for years (an allergy, a profession). Others last an afternoon (the current trip). Others change because of what the search engine just showed: a user who sees a hotel's price discovers they care about budget. A system that never forgets keeps acting on preferences the user has dropped. A system that forgets on a clock, the standard remedy in the preference-drift literature, eventually discards the allergy along with the trip.

We propose that a conversational retriever's memory of the user should forget according to events, not elapsed time. Each profile statement carries its own state: active, suspended because the user switched tasks, or retired because the user revised it. Statements change state when a contradiction, a task pivot, or a reaction to a retrieved item occurs, never because a fixed number of turns has passed. The claim we will test is narrow. On conversations where profile facts are revised or temporarily set aside, this per-statement, event-triggered memory should avoid a trade-off that any single global forgetting rate cannot escape. That trade-off is between violating enduring constraints and obeying stale ones. It should also cost nothing on conversations without drift.

What exists and what is missing

The personalization side comes from TREC iKAT. TREC iKAT 2023 and TREC iKAT 2024 run multi-turn search over about 117M ClueWeb22-B passages, conditioned on a Personal Text Knowledge Base (PTKB), a list of about 17 narrative statements about the user. Large language model (LLM) query rewriting nearly matches manual runs. However, turns that depend on PTKB statements are consistently the weakest, and the PTKB is frozen before the conversation starts. This motivates our work: personalization is already the bottleneck, and the benchmark cannot even express profile change.

A Personalized Dense Retrieval Framework for Unified Information Access attends over a user's last $N$ interactions and reports that personalization hurts users whose behavior shifts. This is direct evidence that a fixed recency window confuses drift with identity.

The drift side supplies the method we argue against. Right Now, Wrong Then (NS-DPO) discounts each old preference by $\gamma^{T-t_i-1}$ and keeps reward accuracy above 60–70% where standard DPO falls to about 50–55%. The discount is global and indexed by time, and the work concerns text generation with no retrieval. We take it as the strongest principled baseline and test whether its single rate is the wrong shape for profile memory.

Three papers constrain the design. AcCoRD shows that "triggered" preferences, which arise only after the agent exposes an item attribute, drive frontier LLM agents down to scores as low as 0.09. That paper evaluates prompting, not retrieval. Cold-Start Personalization via Bayesian Adaptive Questioning (CAPE) elicits preferences upfront and assumes they then stay fixed, which triggered preferences break. Evaluating Rule-based Programming and Reinforcement Learning found that users whose preferences evolved preferred inspectable rules over an opaque reinforcement learner. We therefore keep memory as explicit, readable statements with visible states, not a latent vector.

Two papers supply technique. Context-Aware Drift Detection separates benign context shifts from true concept change by comparing only within matched contexts. We borrow that distinction, but apply it per statement inside a conversation rather than as a batch hypothesis test on image features. MiCRo routes among preference heads online. Improving Context-Aware Preference Modeling shows that conditioning on explicit context sharply improves preference prediction: 98.5% versus 90.1% for GPT-4 Turbo on preference-reversal tests. It also assumes the context is static. Our memory decides which context is currently true, which is the step CAPM leaves out.

Finally, Temporal Preference Optimization for Unsupervised Retrieval makes dense retrievers sensitive to calendar time in the corpus. That is document-side drift, not user-side, so it is complementary and out of scope here.

Core Claim

Let the profile at turn $t$ be statements $s_1,\dots,s_K$, each with a state $z_k(t) \in \{\text{active}, \text{suspended}, \text{retired}\}$. Retrieval conditions only on active statements:

$$ \text{score}(d \mid q_t, \text{profile}) = f\big(d,\ \text{rewrite}(q_t, \{s_k : z_k(t)=\text{active}\})\big), $$

where $q_t$ is the current utterance, $d$ a passage, $f$ an unchanged retriever and reranker, and $\text{rewrite}$ the LLM query rewriter.

State transitions are produced by a small classifier. It reads the statement, the new utterance, and the passages just shown, and predicts one of three things. The first is "no event." The second is "task pivot," which suspends task-scoped statements and restores them if the user returns to that task. The third is "revision," which retires the statement and adds the new one.

Elapsed turns appear nowhere in this rule. The falsifiable prediction concerns any global decay $\gamma$, or any window of the last $N$ turns. On drift conversations, sweeping that knob traces a frontier between two error rates. One is the rate at which retrieved passages violate an enduring constraint. The other is the rate at which they obey a retired one. The event-driven memory should land strictly inside the region that frontier cannot reach. If it lands on the frontier, the idea is wrong and tuned decay is enough.

Research Program

Benchmark. The first build is a benchmark, because none exists. We construct iKAT-Drift from the 2023 and 2024 iKAT topics by editing the persona mid-conversation in three controlled ways.

The first edit is an explicit revision ("actually I've stopped being vegetarian"). The second is a task pivot and return, where an unrelated sub-task is inserted and the original thread is then resumed. The third is a triggered preference in AcCoRD's sense: the inserted user turn reacts to an attribute of a passage the system actually retrieved, such as a price or a distance. Every edited conversation also contains at least one enduring statement that must stay in force throughout.

Relevance after the edit must reflect the updated profile. We judge new top-ranked passages with an LLM judge given the post-edit profile. Human annotators validate a stratified sample, and we report agreement before trusting any comparison. Judgments already in iKAT are reused for unaffected turns.

Memory classifier. The second build is the state-transition classifier. It is a few-hundred-million-parameter cross-encoder trained on synthetic drift trajectories. These are generated with an LLM user simulator in the AcCoRD style, so their labels are known by construction. Training conversations are drawn from topics disjoint from evaluation.

Analysis. The analysis asks where errors come from. We separate failures of the classifier (wrong state) from failures of the rewriter (right state, bad query) by running an oracle-state condition. This mirrors CAPM's decomposition of reward error into a context-inference term and a preference-prediction term.

Baselines

The naive reference is the strongest iKAT-style pipeline applied unchanged to iKAT-Drift. It is an LLM rewriter given the full conversation and the original PTKB, followed by sparse and dense retrieval and a cross-encoder reranker, matching the description of the top automatic iKAT 2024 run. On unmodified iKAT 2024 that run reports nDCG@5 of 0.5156, and the best manual run reports 0.5418 (TREC iKAT 2024). We first reproduce a number near 0.5156 to confirm our pipeline is sound. We found no reported result on Lacuna for any system on a drifting-profile variant of iKAT, so all drift-condition numbers are new measurements.

The strongest principled competitor is NS-DPO-style global discounting moved into retrieval. Each PTKB statement and history turn is weighted by $\gamma^{\text{age}}$ in the rewriter prompt, and low-weight items are dropped below a threshold. The recency-window baseline from Unified Information Access is the same pipeline restricted to the last $N$ turns. For fairness, $\gamma$ and $N$ each get a full sweep on the development split. This is a larger tuning budget than our classifier's single threshold, and we report the whole frontier rather than one point.

The most dangerous competitor is LLM-edited memory. After every turn, the same frontier LLM rewrites the PTKB itself: adding, deleting, and editing statements with a prompt that explains suspension versus revision. It is given the same development examples our classifier sees, as few-shot demonstrations.

Two ablations isolate the key ingredient. The first collapses the three states to two (active/retired), removing suspension, to test whether task-pivot handling matters. The second makes transitions time-triggered with per-statement learned half-lives, to test whether events or per-statement rates do the work. An oracle-state run gives the ceiling.

All systems share the same corpus index, retriever, reranker, rewriter LLM, and judge. The bar for "beating" each competitor is set per comparison. Against the decay and window baselines, we must land strictly outside the swept frontier on both error rates, with confidence intervals bootstrapped over conversations, and stay within one nDCG@5 point of the reproduced baseline on undrifted iKAT. Against LLM-edited memory, we must match its post-edit nDCG@5 while making fewer erroneous deletions of enduring statements. A tie at substantially lower cost per turn would also count.

Why not just…?

Why not just give a frontier LLM the whole conversation and tell it the latest statement wins? That is our naive baseline, and the evidence against it is substantial but indirect. AcCoRD reports that ReAct-prompted frontier models satisfy all preferences in under 30% of scenarios and score as low as 0.09 on triggered preferences. CAPM finds that general LLMs are context-lazy, falling back to their priors when the given context conflicts with them. Those results concern agents and reward models, not retrieval, so iKAT-Drift's post-edit nDCG@5 and stale-constraint violation rate are what settle the question here.

Why not just tune a global forgetting rate or a recency window? This is the main hypothesis under test, not something we dismiss. UIA's degradation on shifting users suggests windows confuse exploration with identity. NS-DPO's single $\gamma$ cannot by construction keep an allergy while dropping a finished trip. If the swept frontier reaches our operating point, we will report that tuned decay suffices.

Why not just let the LLM rewrite the profile after each turn? This is a real threat, and it may well match us on revisions. Our expectation is that it errs on task pivots, deleting statements that should only be paused, and on enduring constraints the user did not mention recently. That expectation is a guess, not a result. The pivot-and-return conversations and the enduring-constraint violation rate are designed to reveal it. If LLM editing wins, the contribution reduces to the benchmark and the error analysis, and we will say so.

Why not just learn online from clicks? Sarkar et al. found that binary-feedback reinforcement learning became erratic and less satisfying when preferences evolved mid-interaction: median satisfaction of 4–5/7 versus 7/7 for rules. CAPE reports that RL needs three to five times more interactions than structured belief models. A single conversation offers too few clicks for online learning to recover from a revision within the next turn or two, so we do not run it as a main baseline.

Proposed Experiments

The primary experiment runs all systems on iKAT-Drift. It reports nDCG@5 on post-edit turns, the enduring-constraint violation rate, and the stale-constraint violation rate, each broken down by edit type.

The decisive check is the two-error frontier plot. The $\gamma$ and $N$ sweeps trace curves, and the event-driven memory, the LLM-edited memory, and the oracle appear as points. The idea survives only if the event-driven point lies outside the curves. A second check runs unmodified iKAT 2023 and 2024 to confirm that the memory does no harm without drift, measured against the reproduced 0.5156 nDCG@5 reference on 2024.

The third check transfers the classifier, without retraining, to AcCoRD-Shop. There the agent retrieves over the 1.2M-product catalog and triggered preferences are native rather than injected. We compare triggered-preference scores against the same prompted agent with and without the memory, using AcCoRD's reported floor of 0.09 for some models as context, not as a matched baseline.

The final check is the oracle-state run. It tells us whether remaining errors lie in deciding what the user currently wants or in turning that into a query. That determines whether follow-up work belongs in the memory or in the rewriter.