When a shopping or travel assistant shows a user a hotel with a rooftop pool, the user may suddenly decide they want one. They did not hold that preference before the agent surfaced it. Current agents track user wishes as if the user were a fixed target observed through noisy messages, so they cannot tell a preference the user always had from one their own suggestion just created. This proposal builds a small recurrent preference tracker for a language-model agent. A recurrent tracker is a module that carries a fixed-size memory forward from turn to turn. Its defining input is the agent's own previous action, meaning which options and attributes it just showed the user, fed back alongside the user's reply. The tracker keeps a compact, fixed-length belief about which preferences are currently active, which are latent but triggerable, and how confident it is. This belief is handed to the agent at every turn without the history of profile text growing.
The opportunity is concrete. On the AcCoRD benchmark, frontier agents reach a "Perfect Outcome," satisfying every user preference, less than 30% of the time. Their scores on triggered preferences, which appear only after the agent surfaces an attribute, fall as low as 0.09, while underspecified requests score 0.7–0.9 (AcCoRD). Our bet is that this gap comes from a missing input rather than missing capacity. The agent's own exposures are a cause of the user's preference changes, and the preference state should be updated as a function of them.
Core Claim
The old idea of feeding a network's previous decision back as an input comes from Links Between Markov Models and Multilayer Perceptrons. That paper showed that a network given its own previous state decision alongside the current observation can estimate the posterior directly, without hand-built transition matrices. That idea is the right tool for preference drift that is partly self-induced. Our claim is testable. A preference tracker whose update conditions on the agent's past exposures will beat an otherwise identical tracker that sees only user utterances, and the gain will concentrate on triggered preferences. The tracker should also latch a triggered preference, holding it even when the user stops mentioning it. This contrasts directly with recency discounting, which fades old evidence uniformly with clock time.
Prior work handles each half separately but not the closed loop between them. S2PNM runs a GRU, a standard gated recurrent network, over a user's passive item history and adds a dynamic preference vector to a static one. It never sees what the system itself exposed, so it treats drift as exogenous. Non-Stationary DPO discounts older preference pairs by $\gamma^{T-t-1}$ during offline training and leaves the deployed model without memory within a session. RLPA tracks a user by emitting a JSON profile as text each turn and trains with profile and response rewards. It is the strongest method for drift inside a dialogue, but its state is conditioned on whatever the model chooses to write, and it grows the context. Dynamic TMoE shows in numerical forecasting that a GRU-based router beats memoryless routers under regime shifts. We take that as method-level support for recurrent state under drift, though not evidence about human preferences.
Research Program
The agent backbone is Qwen-2.5-3B-Instruct, chosen so we can compare directly with RLPA. Beside it sits a tracker: a small GRU whose hidden state is decoded into a per-attribute belief. For each attribute in the task schema, the belief records whether it is currently required, its preferred value or range, the tracker's confidence, and whether it is still latent. At turn $t$ the tracker takes three inputs: its previous state, an encoding of the agent's last action (the products or itineraries shown and their attribute values), and an encoding of the user's reply. The new state is projected into a fixed number of soft prefix tokens, which are learned input vectors the agent's language model reads at each turn. Context cost therefore does not grow with dialogue length. For inspection and for the text baselines, we also decode the belief into a short readable profile.
Training has two stages. First, the tracker is trained with supervised learning against the simulator's ground-truth active-preference set at each turn. AcCoRD formalizes users as a partially observable process with hidden preference specifications, so we expect this signal to be extractable from its simulated users. If it is not, we build an equivalent simulator over WebShop's 1.18M products (WebShop) with scripted triggered and drifting preferences. Second, the tracker and agent are fine-tuned jointly with PPO, using an outcome reward plus a belief-accuracy term, mirroring RLPA's two-level reward so the training signal is matched.
The analysis targets two failure modes the file documents. The first is over-asking: AcCoRD found that prompting agents to flag uncertainty sends weaker models into repeated clarification loops. Our tracker's confidence lets us measure whether the agent asks only when belief is genuinely uncertain. The second is latching versus decay. We probe whether particular hidden units behave like the bistable "flip-flop" self-loops described in The Recurrent Cascade-Correlation Architecture, switching on at a trigger and staying on.
Baselines
The naive reference is the same 3B backbone run with ReAct prompting and the full dialogue history in context, with no tracker. Alongside it we report the published frontier results from AcCoRD: Perfect Outcome below 30% for Claude-4.5, and triggered-preference scores as low as 0.09. We re-run UncReAct, which prompts the agent to emit friction tokens before acting, on our backbone to measure over-asking.
The strongest existing competitor is RLPA, run from its released code (XingYuSSS/RLPA). It is the only method in the file trained specifically to track evolving preferences across turns, and it reports 73.38 average alignment on vanilla ALOE against 44.32 for supervised fine-tuning and 45.27 for DPO. On extended ALOE with unseen attribute types it scores 52.74, against 54.66 for GPT-4o. RLPA has not reported results on AcCoRD, so we retrain it on our simulator episodes with the same backbone, the same number of PPO steps, and the same hyperparameter search budget. Beating it means two things. On AcCoRD's triggered-preference category, we need a statistically significant gain over seeds and an overall Perfect Outcome rate at least as high. On ALOE, we must not fall below 73.38 on the vanilla set.
Two ablations isolate the key ingredient. The first, tracker without exposure feedback, is identical except that the agent's previous action is zeroed out of the tracker's input. The second, RLPA plus exposure log, appends a compact list of shown attributes to RLPA's text profile. This tests whether exposure conditioning helps whether the state is latent or textual. A recency-decayed tracker replaces the learned update with evidence weighted by $\gamma^{T-t-1}$ in the style of NS-DPO, with $\gamma$ tuned on validation data. Finally, a hand-written Bayesian filter keeps per-attribute beliefs, uses a drift prior, and includes an explicit "exposure may trigger" likelihood term. It receives the same structured inputs and is tuned with the same budget.
Why not just…?
Why not just keep the full history in a long context window? The exposures are already in the transcript, so a capable model could in principle reason about them. Two pieces of evidence argue against relying on this. AcCoRD's frontier models had the whole history and still scored as low as 0.09 on triggered preferences. RLPA reports that its SFT and DPO baselines degraded after about 5 turns. Our naive baseline measures this on the same backbone. The counterfactual test described under Proposed Experiments checks whether the long-context model's beliefs actually respond to exposures at all.
Why not just use RLPA's text profile and add the exposures to it? This is the most serious threat, and we treat it as a co-equal hypothesis rather than a strawman. If RLPA plus exposure log matches our tracker, the finding is that exposure conditioning is what matters and the latent recurrent state adds nothing but compactness. We would report it that way. The latent state should win on two fronts if it wins at all. It should hold up better at 50 or more turns, where the text profile grows the context and RLPA's slot schema has to absorb unseen attributes; extended ALOE is where RLPA already trails GPT-4o. It should also latch more reliably after long stretches of unrelated turns.
Why not just tune a recency discount, as NS-DPO does? Discounting fits gradual drift in what a population of users prefers, and NS-DPO beats DPO there; for example, its AlpacaEval length-controlled win rate is 8.93 against DPO's 7.29. A triggered preference behaves differently. It should persist once it appears, and uniform decay gives it the same weight as stale evidence. The recency-decayed tracker baseline and the latching test show directly whether decay erodes triggered preferences that a learned update keeps.
Why not just use a Bayesian filter? With structured attributes, a hand-written filter with an exposure-trigger term may perform well, and it is easier to interpret. We include it with the same inputs. If it matches the learned tracker, the learned component is unjustified for structured domains. The learned tracker could still matter where attributes are fuzzy, such as ALOE's open-ended personas, and we evaluate there separately.
Proposed Experiments
The main evaluation runs on AcCoRD-Shop and AcCoRD-Travel. We report Perfect Outcome rate, per-category preference satisfaction (hard and soft underspecification, unachievable, triggered), task completion, and clarification turns per episode, averaged over at least five seeds. Long-horizon drift and generalization to unseen attributes are measured on vanilla and extended ALOE. We use WebShop success rate and score as a sanity check that the tracker does not hurt plain instruction following; the best published IL+RL agent reaches 29.1% success, against 59.6% for human experts.
Three checks decide the claim. The first is the exposure ablation, restricted to triggered preferences. If removing the agent's own action from the tracker's input does not significantly lower triggered satisfaction, the central hypothesis fails.
The second is a counterfactual replay. We hold the user's replies fixed and swap which attributes the agent had shown, constructing pairs where the same reply means a new triggered preference in one case and a pre-existing one in the other. We then measure whether each method's belief changes appropriately. A tracker that ignores exposures cannot separate these pairs, by construction.
The third is a latching probe. We insert 10, 30, and 60 unrelated turns after a trigger and measure how long the preference is retained. We expect the learned tracker to hold flat while the recency-decayed tracker declines with gap length. Either way, we also check whether any individual hidden unit behaves like a latch.
If the RLPA plus exposure log baseline equals our tracker on all three checks, we will conclude that the field should add exposure feedback to its existing text-profile methods, and that a new recurrent module is not needed.
References
- AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
- Links Between Markov Models and Multilayer Perceptrons
- Modeling Dynamic User Preference via Dictionary Learning for Sequential Recommendation
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
- Dynamic TMoE: A Drift-Aware Dynamic Mixture of Experts Framework for Non-Stationary Time Series Forecasting
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
- The Recurrent Cascade-Correlation Architecture