Drift That Matters: Tracking User Preferences by Which Policy They Select

When a system serves people whose priorities shift, such as a dialogue assistant balancing brevity against thoroughness or an agent balancing speed against energy, the usual approach is to estimate each user's weighting of the objectives and keep re-estimating it as new feedback arrives. Most of that effort is wasted. For any fixed set of candidate policies, a large region of possible weightings all pick the same best policy. A user's weights can wander inside that region indefinitely without changing what the system should do. Meanwhile, much of the apparent change in human feedback is plain inconsistency. In a longitudinal study, people gave different answers to identical questions 6–20% of the time (Moral Change or Noise?). We propose to track each user's preferences at the level of which policy is optimal for them, not at the level of the weights themselves. The system then switches policies only when there is evidence that the user's weights have crossed into a region where a different policy wins.

The geometry is what makes this practical. The returns achievable by an agent's policies form a structured polytope (a many-sided convex shape whose corners are deterministic policies). The best response to any weighting lies on its upper convex boundary, the convex coverage set. Each corner of that boundary therefore owns a cone of weightings for which it is optimal. A standard noisy-choice model makes a sharp, testable prediction: a user whose weights have not moved should flip their answers mostly on comparisons whose options score nearly the same under those weights, which means near the walls between cones. Flips far from any wall are evidence of real change. This gives a principled, per-user way to separate noise from genuine drift, which the literature names as an open problem but has not solved.

Core Claim

Fix a convex coverage set of policies with return vectors $v_1,\dots,v_K \in \mathbb{R}^m$ over $m$ objectives. The optimality cone of policy $k$ is the set of weightings on the probability simplex for which $v_k$ scores at least as high as every other $v_j$. Assume a user with current weighting $w$ and personal noise scale $\beta$ chooses option $a$ over $b$ with probability

$$p_{ab} = \sigma\big(\beta\, w^\top (v_a - v_b)\big),$$

where $\sigma$ is the logistic function. If $w$ is stationary, the chance of a flip on a repeated comparison is $2p_{ab}(1-p_{ab})$. This value peaks when $w$ lies on the hyperplane $w^\top(v_a-v_b)=0$ and decays with distance from it. Our claim has three parts. First, per-user flip rates in real longitudinal data concentrate near these indifference hyperplanes. Second, a detector that asks whether the user has left their current optimality cone, and tests flips against this margin-dependent prediction, achieves lower dynamic regret with far fewer harmful policy switches than methods that decay old data uniformly or track $w$ continuously. Third, the per-user noise scale $\beta$ is the variable that separates the "Simplifier" and "Drifter" behavioral profiles. It should therefore be estimated per user, not pooled.

The closest prior work treats these pieces separately. Learning What Matters Now infers a drifting $w_t$ with a recurrent variational encoder and acts through an envelope Q-network, but it tracks the weight vector itself, assumes a single user, and has no noise-versus-drift test. Right Now, Wrong Then discounts all older preference data by $\gamma^{T-t_i-1}$ in a scalar objective. It cannot tell a decision-irrelevant change from a decision-relevant one, and its decay rate depends on an unknown variation budget. Near-Minimax MORL under Predictable Adversarial Preferences proves that the convex hull of returns is the right operational object under drifting preferences, but it assumes the preference vector is revealed each episode; we infer it from noisy comparisons. MAP maps target palettes to scalarization weights through a convex dual, but it treats the target as static within a run. Moral Change or Noise? motivates the whole proposal: it measured response instability and model instability and found 5–16% higher prediction error for unstable participants. That paper is descriptive and offers no mechanism for acting on the distinction.

Research Program

The first component is the geometry, which we take from prior work rather than invent. In tabular multi-objective environments we compute the exact convex coverage set and its optimality cones using the distance-one neighbor traversal of How to Find the Exact Pareto Front for Multi-Objective MDPs?. That paper shows adjacent boundary corners differ in one state-action entry. Cone walls are therefore cheap to enumerate, and each wall corresponds to a single interpretable policy change. In continuous control, we approximate the coverage set with a finite library of policies trained by entropy-regularized Pareto Q-learning (Multi-Objective RL: Convexity, Stationarity and Pareto Optimality), with optimality cones computed over the library's estimated return vectors. For language models, the library is a set of checkpoints along a helpfulness–harmlessness front produced by MODPO. Our method sits on top of any such library and selects among its members.

The second component is per-user inference. Each user has a posterior over $(w,\beta)$. A population prior is shared across users, in the spirit of the shared/individual split in P-ShareLoRA, but applied to a small number of interpretable parameters rather than adapter matrices. From this posterior we derive the probability that the user's weighting lies in each optimality cone. The deployed policy is the cone with the highest posterior mass.

The third component is a margin-calibrated change test. We run a sequential likelihood-ratio test between two hypotheses: "the user's weighting is stationary inside cone $k$" and "the user's weighting has moved to a neighboring cone." Flips are scored against $2p(1-p)$ under the current posterior. A flip on a near-indifference comparison costs the stationary hypothesis almost nothing, while a flip on a high-margin comparison costs it a lot. Because the alternatives are restricted to neighboring cones, the test has power exactly where switching would change behavior. When the test fires, the posterior is re-anchored on recent data. Otherwise, all history is retained, which avoids the preference inertia versus forgetting trade-off that uniform decay forces.

The fourth component is query selection, to be studied only after the passive version works. When the system can ask for a comparison, it picks pairs whose separating hyperplane is a wall of the current cone, because those are the most informative about decision-relevant movement. This partially addresses the active-exploration gap that AcCoRD exposes. We claim nothing about preferences the user has never formed.

Baselines

The naive reference point is static linear scalarization: weights are fit once per user from their first session and never updated. Learning What Matters Now reports that fixed-weight baselines sit near 0% success in its Maze environment. This baseline mainly tells us how much drift each environment actually contains.

The strongest existing method for the sequential-control setting is Dynamic Preference Inference. It uses a recurrent encoder over a sliding window of transitions to produce a posterior over $w_t$, with an envelope operator for action selection. The paper reports a 59.04% Maze success rate and a 14.85% higher Queue success rate than its best baseline. We run its released configuration on the same environments and drift schedules, using the same policy library as our method for action selection. This isolates the inference component. Both methods get the same number of hyperparameter trials. Beating it means matching or exceeding its success rate and return while lowering dynamic regret and switch count under human-calibrated noise. Human-calibrated noise is a regime the original paper did not test, and it is where we expect the difference to appear.

The strongest method for the preference-data setting is NS-DPO-style exponential decay. On UltraFeedback with mid-dataset flips, it reports reward accuracy "above 60-70%" versus about 50–55% for standard DPO, and an AlpacaEval 2.0 length-controlled win rate of 8.93 versus 7.29. We apply its decay both to per-user weight estimation and to checkpoint selection on TV-HH. The decay rate $\gamma$ is tuned on a validation split with the same budget we give our test's threshold. Beating it means higher held-out choice accuracy and lower regret without per-environment tuning of the drift rate.

A generic change-point baseline runs Bayesian online change-point detection on the per-user logistic weights, with no knowledge of cones. This is the ablation that matters most. It shares our noise model and per-user structure but drops the geometric relevance criterion. A second ablation keeps the cones but replaces the margin-calibrated flip model with a constant flip rate. It tests whether conditioning on margin is doing the work. A third ablation pools $\beta$ across users, which tests the subjective-variance component.

Why not just…?

Why not just use exponential recency weighting, as NS-DPO does? It is simple, and Right Now, Wrong Then shows it beats unweighted training under drift. Its weakness is structural: one $\gamma$ discounts everything, so it forgets stable constraints at the same rate as it forgets stale ones. Its optimal $\gamma$ also depends on an unknown variation budget. This alternative is a real threat when drift is frequent and large, because then most changes do cross cones and decay is nearly right. Our experiments sweep drift magnitude for exactly this reason. We expect an advantage only when drift is small or mostly within cones and noise is substantial. If that regime turns out to be empirically rare in the kidney data, the proposal's practical value shrinks, and we will report that.

Why not just track $w_t$ continuously with a learned encoder and take the best policy for the current estimate? That is essentially what Learning What Matters Now does, and with abundant clean feedback it may be just as good. The difference shows up under noise. A continuous tracker that moves its estimate after each flip will cross cone walls spuriously and switch policies, while ours requires margin-weighted evidence before switching. The switch-count and regret comparison under human-calibrated flip rates settles this directly. If the encoder matches us there, the cone machinery is unnecessary.

Why not just run off-the-shelf change-point detection on the per-user weights? This is the most serious alternative, so it is built in as the main ablation. Parameter-level detection raises alarms whenever $w$ moves, including moves that change nothing. It can also miss a small move that happens to cross a nearby wall. If the geometry-free detector matches ours on regret and false switches, then the polytope contributes only bookkeeping, and the honest conclusion is that per-user noise calibration, not geometry, carries the result.

Proposed Experiments

The first and decisive check uses the longitudinal kidney-allocation data from Moral Change or Noise?, with more than 400 participants over 3–5 sessions. Here the options are patients described by eight features. Their feature vectors play the role of return vectors, and each user's per-session logistic weights define a point in weight space. Using weights fit only on earlier sessions, we compute each repeated pair's margin $w^\top(v_a-v_b)$. We then test whether test-retest flips in later sessions follow $2p(1-p)$ as a function of that margin. If flips are flat in margin, the noise model is wrong and the method has no foundation; we would stop or revise. If the prediction holds, we next ask whether cone-aware forecasting of the next session's choices beats per-session refitting, exponential decay, and parameter-level change-point detection, reporting accuracy separately for each behavioral cluster. The predicted signature is large gains for the Drifters and the Simplifiers' feature-dropping, which appears in our model as $w$ moving onto a lower-dimensional face. No reported number was found on Lacuna for any method's next-session accuracy on this dataset beyond the 5–16% error gap between clusters, so all baselines are run by us.

The second experiment is controlled sequential decision-making. We fit per-cluster distributions of session-to-session weight change and of $\beta$ from the kidney data. We use them to drive simulated users in Deep-Sea-Treasure-style tabular problems, where the exact coverage set is available (How to Find the Exact Pareto Front), and in the Queue, Maze, and HalfCheetah environments of Learning What Matters Now. Feedback is pairwise comparisons only, never reward vectors, which also relaxes the vector-feedback assumption of Near-Minimax MORL. The metrics are dynamic regret in scalarized return under the user's true current weighting, number of policy switches, detection delay and false-alarm rate for true cone crossings, and success rate where the environment defines one. We sweep drift magnitude and noise level, mapping out where each baseline wins.

The third experiment transfers the method to language models. On TV-HH from Right Now, Wrong Then, we build a helpfulness–harmlessness checkpoint library with MODPO. Each simulated user's comparisons follow drift trajectories calibrated as above. Our method selects a checkpoint per user over time, and we compare it with NS-DPO-trained policies and decay-based checkpoint selection. Metrics are held-out preference accuracy, reward-model scores on both axes, and switch count. A tie with NS-DPO here would still count as useful if it comes with fewer switches and no tuning of the drift rate. A loss would indicate that finite checkpoint libraries are too coarse for the cone abstraction to pay off in generation.