Aging Preferences Toward Ignorance: Opinion-Geometry Belief Tracking for Agents Under Preference Drift

Conversational agents that shop, book travel, or recommend on a user's behalf have to keep track of what the user wants while those wants change. Most current systems store this as one number per preference: a reward score, a logit, or a probability. That number cannot tell apart two very different situations. In one, the user has said they don't care between two options, which is indifference. In the other, the topic has never come up, or the user's old statement no longer holds, which is ignorance. We propose a small belief tracker for each preference dimension that keeps these two situations separate. Each dimension is stored as a subjective-logic opinion, a point in a triangle whose corners are "believes," "disbelieves," and "doesn't know." When new evidence contradicts an old confident opinion, the tracker moves that opinion toward the "doesn't know" corner instead of flipping it or averaging it away. The agent then asks a question, or shows a contrasting option, only when a dimension is both uncertain and able to change which item it would pick.

This matters because the clearest benchmark of preference dynamics shows agents failing in two opposite directions. In AcCoRD, which formalizes user–agent collaboration under shifting preferences as a sequential decision problem with hidden user state, frontier models satisfy every preference in fewer than 30% of episodes. On triggered preferences, which surface only when the user sees a new option, scores fall as low as 0.09. Meanwhile, weaker models prompted to flag uncertainty over-ask and use up their turn budget. Our hypothesis is that missed triggered preferences and over-asking are one error seen from two sides: the agent confuses indifference with ignorance.

Core Claim

A per-dimension belief state with ignorance as its own coordinate will satisfy more preferences with fewer questions than prompting-based agents and than fixed-rate forgetting. The two ingredients we expect to matter are these. First, drift is handled by aging that is triggered by conflict and applied separately to each dimension. Second, questions are gated by decision relevance.

We are explicit about what is not new. A binary subjective-logic opinion is equivalent to a Beta distribution. Moving toward the uncertainty corner is equivalent to discounting evidence counts. The contribution is therefore not a new probability calculus. It is three things, each of which can be tested in isolation:

  1. Where the aging rate comes from. Conflict sets it, separately per dimension, instead of a single global rate.
  2. How the agent uses the ignorance coordinate. It decides when to ask.
  3. How it fuses evidence from sources of different reliability. This uses the geometric operators, which are known to behave well.

Research Program

Representation. The state for a user is a set of opinions $\omega_k=(b_k,d_k,u_k)$, one per preference dimension $k$. A dimension is, for example, a product attribute such as material, price band, or color in the WebShop catalog underlying AcCoRD-Shop. The components satisfy $b_k+d_k+u_k=1$ and come with a base rate $a_k$.

Unmentioned dimensions start at the ignorance corner, $u_k=1$. Their base rate comes from how often the attribute matters in the product category, rather than from the fixed $a=0.5$ that existing barycentric implementations assume. This directly relaxes one limitation of the geometric literature. The expected preference $b_k+a_ku_k$ is what enters the agent's item scoring.

Evidence and fusion. Each user turn is parsed by the shared language model into evidence about specific dimensions, tagged by source:

  • an explicit statement,
  • an implicit rejection of a shown item, or
  • the model's own inference.

Lower-reliability sources are discounted with Cerutti et al.'s geometric discount operator. That work treats discounting as a linear projection toward the uncertainty corner inside an "admissible region," so an unreliable report defaults to ignorance rather than to false disbelief. Discounted reports are then combined with their centroid fusion operator. On synthetic trust networks those operators improved geometric accuracy by about 56% over Jøsang's algebraic operators, with more regular error distributions. A companion study confirms this.

That evidence is entirely synthetic, from 50 simulated agents. It motivates the choice of operators but does not validate them on human dialogue, and testing them on human dialogue is part of what we do. Subjective Logic Encodings supplies the precedent for using discounting plus fusion as a learning target. It does this for crowd labels with a stationary truth; we apply it to a target that moves.

Conflict-triggered aging. Before fusing a new report $\hat\omega_k$ into the current opinion $\omega_k$, we measure their conflict with the standard subjective-logic conflict measure:

$$c_k=b_k\hat d_k+d_k\hat b_k.$$

$c_k$ is high when a confident old belief meets a confident new disbelief, or the reverse. We then age the old opinion toward the uncertainty corner $e_u=(0,0,1)$:

$$\omega_k \leftarrow \omega_k+\lambda_k(e_u-\omega_k),\qquad \lambda_k=1-e^{-\kappa c_k-\rho\,\Delta t_k}.$$

Here $\Delta t_k$ is the number of turns since dimension $k$ last received evidence. The constants are $\kappa\ge 0$, the sensitivity to conflict, and $\rho\ge 0$, a small background forgetting rate, and both are tuned on a development split.

With $\kappa=0$ and a single shared $\rho$, this reduces to uniform exponential forgetting. That is the state-tracking analogue of the fixed decay factor $\gamma$ in NS-DPO, which weights each preference pair at timestamp $t_i$ by $\gamma^{T-t_i-1}$. The comparison between the two is therefore built in. The intended behavior is that a real preference change passes through a visible "unsure again" state, while dimensions that were never contradicted keep their evidence.

Decision-relevant asking. Poupart and Boutilier showed that an approximation error in a belief state costs nothing when the error direction is orthogonal to the gradient of the value function. We use the same logic to decide when uncertainty is worth resolving. Let $m$ be the score margin between the agent's top two candidate items. Dimension $k$ is worth a question, or worth surfacing a contrasting item, only if $u_k\,|\partial m/\partial E_k|$ exceeds a threshold. Here $E_k$ is the expected preference on dimension $k$.

This rule has two effects:

  • A dimension the user is indifferent to has low $u_k$, so it is never asked about, which prevents over-asking.
  • A never-discussed attribute that separates the two leading candidates has high $u_k$ and a nonzero gradient, so it triggers an option-surfacing action. This is our mechanism for triggered preferences.

This is a cheap, local stand-in for the value-of-information query selection in Boutilier's elicitation work. That work assumes a fixed utility and uses expensive continuous optimization. We give up its optimality in exchange for tractability across the full product-attribute space.

Agent. The tracker is serialized into the language-model agent's context as a compact table of dimensions with their $(b,d,u)$ values. The ask/surface rule is exposed as a tool the agent can call. The language model still does parsing, search, and dialogue; it does not have to keep uncertainty in its head.

Baselines

Plain agent (naive reference). The same language model runs as a ReAct agent on AcCoRD with the full dialogue history in context and no explicit state. This is the configuration AcCoRD reports, with Claude-4.5 below 30% Perfect Outcome and triggered-preference scores as low as 0.09. We rerun it with our chosen open model so that numbers are comparable.

Strongest existing uncertainty baseline. UncReAct, AcCoRD's prompting method that emits a <friction> token to pause and ask, is the most direct competitor on the "when to ask" question. Its reported weakness is over-asking. The file gives no separate UncReAct Perfect Outcome figure beyond the under-30% ceiling for frontier models.

Strongest method for dynamic preference weighting. AdaPA-Agent scores preference alignment with a language model and blends preference-conditioned output distributions. It reached 41.45% recommendation success within 3 steps on Reddit-Movie versus 30.20% for ReAct. We run it on AcCoRD with the same backbone. We also run our tracker on Reddit-Movie so that we meet AdaPA on its home benchmark and must at least match 41.45%.

Fairness and the bar to clear. All agents share one backbone, one turn budget, and one simulated-user configuration. Each receives the same tuning budget on a held-out scenario split: prompts for the prompting baselines, and $\kappa$, $\rho$ and the ask threshold for ours. Beating the baselines means a statistically reliable gain in Perfect Outcome on AcCoRD-Shop and AcCoRD-Travel with the same or fewer clarification turns. It also means a gain on the triggered-preference sub-score large enough to be read off AcCoRD's reported 0.09 floor rather than within noise.

Ablations. Four ablations isolate the key ingredients:

  1. Scalar probability. Replace each opinion with a single number, which collapses ignorance into indifference.
  2. Fixed-rate aging. Set $\kappa=0$ with a shared $\rho$, the NS-DPO-style control.
  3. Uncertainty-only asking. Ask whenever $u_k$ is high, removing the decision-relevance gate.
  4. Algebraic operators. Swap the geometric operators for Jøsang's algebraic ones.

Why not just…?

Why not just use a Beta distribution with exponential forgetting? In the binary case this is mathematically what our representation is. We do not claim the triangle beats it; conceding the equivalence is why our claims rest elsewhere. What a standard discounted-Beta filter lacks is conflict-driven, per-dimension forgetting and a decision-relevance gate on questions. The fixed-rate aging ablation is exactly this alternative. If it matches the full system, the conclusion is that ignorance tracking plus the ask rule is what helps, and conflict-triggered aging is unnecessary. We would report that as the finding.

Why not just keep a text scratchpad of known and unknown preferences? This is a real threat, because a language model can maintain structured notes cheaply. AcCoRD's own evidence speaks only to unstructured prompting: ReAct and UncReAct miss triggered preferences and over-ask. It does not test structured memory. We therefore include a scratchpad variant with the same dimension slots but only three labels (liked, disliked, unknown) and no graded values or ask rule. If it closes most of the gap, the gain comes from structure rather than from graded ignorance, and we would say so.

Why not just apply NS-DPO-style recency weighting to everything? A single decay rate throws away stable dimensions together with drifted ones, which is the limitation NS-DPO itself acknowledges. The controlled experiment below tests this directly by drifting one axis and leaving the rest fixed.

Why not just prompt the model with numerical preference weights? AdaPA's comparison against ReAct already shows that stating weights in natural language underperforms explicit distribution blending. Our tracker keeps the numbers outside the model's reasoning and uses them only through scoring and the ask rule.

Proposed Experiments

Main experiment. Run all agents on AcCoRD-Shop and AcCoRD-Travel. We report:

  • Perfect Outcome rate,
  • per-dynamic sub-scores (hard and soft underspecification, unachievable, triggered),
  • clarification turns per episode, and
  • task completion.

The decisive check is two-sided. The full system must raise triggered-preference scores and also cut questions relative to UncReAct. The scalar-probability ablation should lose on at least one of the two. If ignorance and indifference are really being conflated, collapsing them should either miss triggered preferences or bring back over-asking.

As a sanity bound, the underlying WebShop environment reports 59.6% success for expert humans versus 29.1% for the best learned agent. Its option-selection sub-score is 73.9 for humans versus 38.9 for the best model. Option selection is where we expect attribute-level tracking to help most.

Controlled drift experiment. On UltraFeedback's four rated axes, we simulate drift by flipping the preference on only one axis at a chosen change point. We then compare two reward models: one trained with NS-DPO's global $\gamma$, and one whose per-pair weights come from our per-axis conflict-triggered aging.

NS-DPO reports reward accuracy "above 60–70%" under full flips versus 50–55% for DPO. There is no reported result on Lacuna for single-axis drift. Our prediction is that fixed $\gamma$ loses accuracy on the three stable axes, and per-axis aging does not. If per-axis aging shows no advantage there, the adaptive-rate ingredient fails its cleanest test.

Calibration check. We check whether rising $u_k$ actually predicts an upcoming user correction on that dimension. We measure this as the area under the ROC curve of $u_k$ for predicting the next contradiction, computed from simulated-user logs. This tests whether the ignorance coordinate carries real signal before we credit it with the downstream gains.