Is It Drift or Disagreement? Topic-Resolved Hodge Tracking of Preference Change

When people vote between two chatbot answers, the tallies move over time. Preference-learning systems usually read any such movement as "preferences changed" and respond by discounting old data. Some of that movement has other causes. The mix of topics people ask about can shift. New models can enter the comparison pool. A single topic can also split into subgroups of users who want opposite things. Each cause calls for a different fix, and discounting old votes addresses only the first: a real reordering of what people prefer within a topic.

This note proposes a way to tell these cases apart, one topic at a time. We cluster prompts into topics. Within each topic and time window, we break the pairwise comparison graph into two parts using combinatorial Hodge theory. The first part is a consistent ranking. The second is leftover circular conflict, of the form A beats B, B beats C, C beats A. How fast the consistent ranking moves gives each topic its own discount rate, estimated from data. A rise in circular conflict signals user subgroups within a topic, and there the fix is to split the topic, not to forget old data. The claim is testable on the timestamped pairwise votes in LMSYS Chatbot Arena and on controlled drift variants of standard alignment datasets.

Core Claim

Three pieces of prior work each cover part of this question. Right Now, Wrong Then: Non-Stationary DPO (NS-DPO) weights each preference pair by $\gamma^{T-t_i-1}$, a single global exponential decay. It does this inside direct preference optimization (DPO, a method that tunes a language model directly on "answer A beat answer B" pairs). The authors themselves note two limits: the decay cannot tell durable data from transient data, and choosing $\gamma$ well requires knowing the total amount of drift in advance.

Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic clusters Chatbot Arena prompts into 29 topics and finds that model win rates differ sharply by topic. For example, gpt-4-0314 wins 49.97% in "Social Issues and Ethical Dilemmas." However, it pools the whole April–August 2023 collection window into one static snapshot.

Statistical ranking and combinatorial Hodge theory (HodgeRank) splits incomplete pairwise data into a consistent ranking plus circular leftovers. The authors explicitly do not handle subpopulations or streaming updates.

Our claim is that these three pieces fit together into one diagnostic. We make four specific predictions. First, per-topic drift rates in real LLM preference data differ enough that a single $\gamma$ is measurably wrong for some topics. Second, a per-topic discount set from the observed motion of each topic's Hodge ranking, with no oracle knowledge of total drift, matches or beats a grid-searched discount. Third, topics where circular conflict grows over time gain nothing from any discount and gain from splitting. Fourth, part of the apparent global drift is really a shift in which topics people ask about, and a pooled model mistakes it for a change in preference.

Mechanism

For topic $k$ and time window $w$, let $Y_{k,w}$ be the edge flow on the model-comparison graph. Each edge between two models carries the average margin by which one beat the other, weighted by how often that pair was compared. HodgeRank finds the least-squares decomposition

$$Y_{k,w} = \nabla s_{k,w} + R_{k,w},$$

where $s_{k,w}$ is a score for each model (the "gradient," or consistent ranking) and $R_{k,w}$ is the curl-plus-harmonic residual (circular inconsistency). From this we define two quantities:

  • the cyclicity $\rho_{k,w} = \|R_{k,w}\|^2 / \|Y_{k,w}\|^2$;
  • the drift rate $\delta_k$, the average of $\|s_{k,w+1} - s_{k,w}\|$ across windows after subtracting the change expected from sampling noise alone. We estimate that noise level, $\sigma_k$, by bootstrapping within each window.

The per-topic discount comes from a bias–variance trade-off. Suppose the scores drift roughly linearly at rate $\delta_k$, and the topic has $n_k$ votes per window with noise $\sigma_k$. Then averaging over an effective window of length $L$ has variance about $\sigma_k^2/(n_k L)$ and lag bias about $\delta_k^2 L^2$. The error is minimized at

$$L_k^\ast \propto \left(\sigma_k^2 / (n_k \delta_k^2)\right)^{1/3}, \qquad \gamma_k = 1 - 1/L_k^\ast.$$

Busy, stable topics such as coding get $\gamma_k$ close to 1. Sparse, fast-moving topics get aggressive discounting. When $\delta_k$ is indistinguishable from noise, the rule reduces to $\gamma_k = 1$, the same stationary fallback NS-DPO recovers.

Separately, when $\rho_{k,w}$ rises significantly over time, we mark the topic as fragmenting. We then re-cluster its prompts one level finer and repeat the decomposition. We expect such splits to lower the cyclicity. If they do not, the conflict belongs to individual users rather than to identifiable prompt subtypes, and we report that.

Two features of the Arena setting make this workable.

  • Cheap exact decomposition. HodgeRank's worst-case cost for the curl and harmonic parts is $\mathcal{O}(|V|^9)$, but the "items" here are about 25 models. The full decomposition is therefore trivial to compute for every topic and window.
  • Uneven pair coverage is handled. Arena comparisons are partial and unevenly sampled: each vote covers two of about 25 models, and models enter mid-collection. HodgeRank's least-squares estimate is built for unbalanced comparison designs. Learning to Rank from Incomplete Rankings shows that Borda count and Rank Centrality fail to converge under non-random censoring, which is why we avoid those simpler aggregators. Unlike PIRATE, we do not close the graph transitively. As that paper's setting implies, transitive patching across windows is unsafe exactly when preferences drift.

Research Program

Stage 1: topics and graphs. We run the BERTopic pipeline from Bhandarkar et al. (all-MiniLM-L6-v2 embeddings, UMAP, HDBSCAN, c-TF-IDF). BERTopic is a topic model that clusters sentence embeddings of prompts and labels each cluster with its distinctive words. We run it once on prompts from the first collection month and assign later prompts to the nearest existing topic, so topic identity stays fixed across time. HDBSCAN's noise cluster (Topic -1), which Bhandarkar et al. discard, is kept as its own stream. We test whether topics that emerge later first appear there as rising density.

We then build per-topic, per-two-week comparison graphs from the pairwise votes. One precondition must be verified first: that the released Arena votes carry timestamps fine enough to support two-week windows. If they support only monthly bins, the design coarsens accordingly.

Stage 2: decomposing global drift. We split the change in pooled win rates into two parts. The first is a composition term: topic shares change while within-topic scores are held fixed. The second is a within-topic term: scores change while shares are held fixed. This shows how much of the "drift" a global method would react to is really a change in what people ask. The file does not report any such split, so we have no prior number for it.

Stage 3: use in training. We plug the $\gamma_k$ into NS-DPO's loss in place of the scalar $\gamma$. Fragmenting topics are handled by finer topic splits rather than by discounting.

Baselines

The naive reference point is a pooled static Bradley–Terry model (which assigns each model a strength score and predicts win probabilities from score differences) fit on all past votes. It ignores both topic and time. A second naive reference, the most-recent-window-only model, marks the opposite extreme of "forget everything."

The strongest existing drift method is NS-DPO's global exponential decay. For forecasting we apply it as time-weighted Bradley–Terry. For training we apply it in its original DPO form. Its $\gamma$ is grid-searched on a validation window, using the same number of grid points as any other tuned method. It is the most direct competitor because it is the published answer to exactly this problem. On its own non-stationary benchmark it reports a length-controlled AlpacaEval 2.0 win rate of 8.93, against 7.29 for DPO, on UltraFeedback-LM. It also reports reward accuracy above 60–70% where DPO falls to about 50–55% under preference reversals. Beating it means lower held-out log-loss on future Arena votes, with the gain concentrated in the topics our diagnostic flags as fast- or slow-drifting. In the training experiments, beating it means higher reward accuracy and win rate on the topic-heterogeneous drift benchmarks described below.

The strongest dynamic-ranking competitor is Preference-Based Dynamic Ranking Structure Recognition. It tracks time-varying Bradley–Terry scores with a kernel spectral estimator and detects change points. It reports 1.0000 Kendall τ and full tier recovery on its synthetic data. It has no topic input, so we run it separately within each of our topics with its kernel bandwidth tuned per topic. Beating it on score tracking is not required, since it may track the consistent ranking as well as we do. What it cannot provide is the cyclicity signal. The comparison that matters here is whether the fragmentation diagnostic improves forecasts in flagged topics.

Per-topic grid-tuned decay, with a separate validated $\gamma$ per topic, is the most important control. It asks whether our closed-form rule adds anything beyond per-topic tuning.

Three ablations isolate the key ingredients. The first drops the fragmentation routing and discounts every topic. The second shuffles topic labels while preserving topic sizes, which tests whether the topics' meaning matters or just the partitioning. The third replaces HodgeRank scores with raw per-topic win rates, as in Bhandarkar et al., which tests whether the incomplete-comparison model matters.

All methods see identical votes, windows, and topic assignments, and receive the same validation budget.

Why not just…?

Why not just grid-search a separate $\gamma$ per topic? This is the real threat. With enough votes per topic, a tuned per-topic $\gamma$ could match our rule. If it does, the discounting half of our contribution reduces to a tuning-free shortcut. That is still useful, because NS-DPO names needing the drift budget in advance as a limitation. But it is modest. We report this comparison directly and separately for data-rich and data-poor topics, where grid search should overfit a single validation window. The fragmentation diagnostic is untouched by this alternative: no choice of $\gamma$ fixes a topic whose users disagree.

Why not just use global NS-DPO? Its authors state that one decay cannot separate durable from transient data. Coding-related prompts alone exceed 10% of traffic according to LMSYS-Chat-1M. A $\gamma$ fit to fast topics will throw away that stable signal. Stage 2 also tests whether global NS-DPO reacts to changes in topic mix as if they were preference change. Whether this hurts in practice is exactly what the per-topic error breakdown will show.

Why not just run Lu et al.'s dynamic ranking within each topic? It is strong at tracking the consistent ranking, and we expect near-parity there. However, the file describes it as limited to one-versus-one comparisons, and it models only score movement. A topic whose conflict comes from subgroups with opposite preferences looks to it like noisy scores. Our fragmentation test checks whether splitting such topics lowers forecast error where smoothing cannot.

Why not just give a neural reward model the prompt and timestamp as inputs? This could in principle learn topic-specific drift implicitly. We include it as an extra baseline at matched parameter and tuning budget. Our expectation is that five months of sparse per-topic votes are too little for it to separate drift from noise. Its failures would also be uninterpretable in a way that the gradient-versus-cyclicity split is not.

Proposed Experiments

Synthetic check. We plant three regimes across topics in simulated partial comparisons among 25 items with realistic uneven pair sampling. The first regime is stationary. The second has score drift at varied rates. The third has cyclicity growth, created by adding a second user subgroup with a reversed ordering. The diagnostic must classify the regimes correctly. Discounting must help only in the drift regime, and splitting only in the growth regime. If discounting also "helps" the growth regime, the separation claim fails.

LMSYS Chatbot Arena forecasting. We train on all windows up to $w$ and predict window $w+1$ votes. We report log-loss and accuracy, averaged both per topic and per vote, plus the Stage 2 split. No reported result on Lacuna was found for temporal vote forecasting on this data. The only per-topic figure available is Bhandarkar et al.'s static win rates, which serve as a sanity check on the first window. For comparison with prior HodgeRank results, cyclicity on Netflix was 0.30% in Jiang et al. A decisive negative outcome would be $\delta_k$ confidence intervals that overlap across nearly all topics. That would mean topic-conditioned drift is not present in this data, and we would report it as such.

Training under topic-heterogeneous drift. We assign BERTopic topics to UltraFeedback-LM and TV-HH prompts (the Time-Varying Helpful-Harmless variant of Anthropic HH-RLHF used by NS-DPO). We then flip preferences at different rates and times per topic, and add a subgroup-reversal condition in one topic. We train global NS-DPO, per-topic grid NS-DPO, and our weighting on the same base model with the same compute. We measure held-out reward accuracy per topic and AlpacaEval 2.0 length-controlled win rate, against NS-DPO's reported 8.93 in its own setting. The decisive check is that our method closes most of the gap between global NS-DPO and a per-topic oracle discount without using the oracle. It must also avoid losing accuracy on stable topics, which is where a global discount should forget signal it still needs.