When people label text as toxic, offensive, ironic, or morally wrong, they disagree. A growing body of work treats that disagreement as real signal rather than noise and trains models to predict how a specific person would label a specific text. A separate body of work notices that human preferences change over time and down-weights old feedback so a model tracks what people want now. Each line quietly assumes away the other's phenomenon. Annotator-aware models treat each person as frozen. Drift-aware models treat the whole population as one crowd that shifts together. So when two labels on similar texts conflict, current systems cannot tell whether the cause is who gave them (two people with different thresholds), when they were given (everyone's norms moved), or one person changing (an annotator becoming desensitized, say, over a long campaign).
This proposal builds and tests a model that pulls these three sources apart. Each annotator gets a stable position, the population gets a shared time trend, and each annotator gets a small personal drift term whose size is learned per person and shrunk toward the group. The practical stake is concrete. Uniform time-discounting in preference learning will quietly erase a stable minority view if the people holding it happened to label early. Frozen per-annotator models will be pulled in two directions by the same person's early and late labels. A model that separates the two can keep enduring minority perspectives while still forgetting labels that really have gone stale. The first deliverable, though, is a measurement: whether, and how much, individual annotators drift at all.
Core Claim
Our claim is that the right amount of memory for an old label depends on whose label it is. The rate at which an annotator's own behavior has moved should set it, not elapsed calendar time. We expect a model that learns per-annotator drift rates under a shared prior, alongside a population trend, to beat both frozen annotator models and globally discounted models on forward-in-time prediction of individual labels. We also expect it to preserve accuracy on annotators who hold a persistent minority view, a group that global discounting should specifically harm. If annotators turn out not to drift measurably, the same model should collapse to a static annotator model with no loss. That result would be informative, because it would tell the perspectivist community that ignoring time is safe.
Where This Sits
The motivating evidence comes from both sides. On the perspectivist side, Majority Vote Silences Minority Values shows that HateXplain annotators who disagree often highlight identical words yet choose different labels. That points to differing personal harm thresholds rather than careless reading, though the paper treats those thresholds as constants. Architectures that model individuals inherit the same constancy. ACTOR gives each annotator a fixed classification head on a shared encoder. Hypernetworks for Perspectivist Adaptation generates one fixed low-rank adapter (LoRA, a small trainable weight update added to a frozen network) per annotator ID. Personalized Large Language Models conditions a fine-tuned LLM on a user-ID token. Unifying Data Perspectivism and Personalization averages embeddings of a Reddit user's past comments as an unordered bag. None of them takes a timestamp as input. We reuse their machinery, a shared encoder plus an annotator-conditioned decision function, and add time to it.
On the drift side, Right Now, Wrong Then introduces non-stationary direct preference optimization (NS-DPO). It weights each preference pair by $\gamma^{T-t_i-1}$, so older pairs count exponentially less. The paper itself notes two limits that motivate us. Choosing $\gamma$ requires knowing in advance how much the preferences will change. And every sample from the same time gets the same weight, whatever its source. We keep its insight that smooth discounting beats hard truncation. We change what the discount is attached to. The Realignment Problem handles drift as a top-down policy change and sorts historical pairs with a single proxy judge. That is the opposite of our bottom-up setting, where different people change at different rates.
Two papers act as constraints rather than as motivation. Beyond Demographics finds that demographic attributes help only as stand-ins for individual IDs and give no gain on unseen annotators. So our stable annotator component must be learned from each person's labels, not inferred from demographics. The same HateXplain paper reports that a per-annotator multi-head model collapsed to 0.245 accuracy on contested offensive cases. Adding free per-person time parameters would make that variance problem worse. This is why the drift term is shrunk toward a shared prior and is not left free. Finally, Humans Hallucinate Too reminds us that some within-annotator inconsistency is plain error. Our model must avoid labeling noise as drift, and the evaluation includes a check for exactly that.
The Model
Let $h(x)$ be a shared text encoding, $a$ an annotator, and $t$ the time or position at which the label was given. We write the annotator's effective preference vector as a sum of three parts:
$$\text{logit}\, p(y \mid x, a, t) = f\big(h(x),\; u_a + g_t + d_{a,t}\big), \qquad d_{a,t} = d_{a,t-1} + \epsilon,\; \epsilon \sim \mathcal{N}(0, s_a^2).$$
Here $u_a$ is the annotator's stable position, $g_t$ is a smooth population trend shared by everyone, and $d_{a,t}$ is a personal random walk whose step size $s_a$ is learned per annotator. A hierarchical prior $\log s_a \sim \mathcal{N}(\mu, \tau^2)$ pools information across annotators. Someone with few labels inherits a typical drift rate. Someone whose labels clearly move earns a larger one. Under a random walk, the effective memory length for annotator $a$ comes from the ratio of $s_a$ to label noise, as in a Kalman filter. Each person therefore gets an implied discount factor learned from data, which addresses NS-DPO's unknown-$\gamma$ problem at the individual level. The decision function $f$ can be a head, an adapter generated by a hypernetwork as in the hypernetworks paper, or a user-conditioned LoRA. We use whichever backbone is strongest in the static setting, so any gain comes only from the time structure.
Two identifiability choices matter. The population trend $g_t$ absorbs any movement common to all annotators, so $d_{a,t}$ captures only individual deviation from it. The $u_a$ are centered so that a stable minority annotator shows up as a large constant offset, not as drift. This separation lets the model keep a 2019 label from a persistent minority annotator at full weight while discounting a 2019 label from someone who has since changed.
Research Program
The work proceeds in three stages. The first is measurement. We audit which perspectivist datasets keep the order or timing of each annotator's labels. Reddit's r/amitheasshole verdicts from the social-norms paper carry natural comment times across long user tenures. For DEMO, the Gab Hate Corpus, HS-Brexit and HateXplain, the file does not say whether within-annotator order is retained, and we will check. On every ordered dataset, we test for individual drift by comparing a static per-annotator model with the full model on held-out late labels from each annotator, and we also estimate the posterior over $s_a$. We report the fraction of annotators whose drift rate is reliably above zero. We also check where drift concentrates. The social-norms paper finds that family and intimate conflicts are the most individualized domain, so it is the natural place to look first.
The second stage is controlled evaluation on semi-synthetic data, where the true decomposition is known. Starting from densely annotated corpora, we inject three effects separately and together: a population shift in the labeling threshold, individual threshold drift for a random subset of annotators, and a group of minority-view annotators who are stable but labeled mostly early. The third effect is the case where we predict global discounting fails. We also inject pure label flips with no temporal structure, to measure how often each method mistakes noise for drift.
The third stage carries the idea into preference learning. 2C NSGO, from the NS-DPO paper, simulates opinion migration from US-centric to Japan-centric answers on GlobalOpinionsQA. We build a variant in which a persistent subgroup keeps US-centric preferences throughout while the majority migrates, and we tag pairs with subgroup identity. We then compare NS-DPO's global discount with a DPO loss weighted by the per-source discount implied by our model.
Baselines
The naive reference point is a static annotator-aware model trained on all labels with no time information. On Reddit verdicts this is the history-averaged sentence-embedding model from Unifying Data Perspectivism and Personalization, which reports roughly 86% accuracy. On the hate and offensiveness corpora it is the user-ID LoRA model from Personalized Large Language Models or the adapter hypernetwork. For the hypernetwork, numerical scores were truncated in the source summary, so we will rerun it rather than quote it. A content-only model, with no annotator information, sits below both as a floor.
The strongest existing method for drift is NS-DPO's exponential discount. For preferences, the file reports that it keeps 60–70% reward accuracy under a mid-dataset flip on TV-HH where standard DPO falls to 50–55%. It also reports an AlpacaEval 2.0 length-controlled win rate of 8.93 against 7.29 for DPO on UltraFeedback-LM (Right Now, Wrong Then). For 2C NSGO there is no reported result on Lacuna, so we will rerun NS-DPO there. For classification, we port the same global discount onto the annotator-aware loss. This makes it the direct "when but not who" competitor.
Three further baselines target the proposal's key ingredient. The first gives each annotator their own discount, with $\gamma$ tuned per annotator on a validation split, or with a shared grid for annotators with few labels. The second adds a time token to the user-ID LLM and lets it learn person-by-time interactions implicitly. The third keeps a per-annotator rolling window of the most recent $K$ labels. The ablations remove $g_t$, remove $d_{a,t}$, and fix $s_a$ to a single shared value. The last ablation isolates whether per-person drift rates, rather than drift as such, carry the gain.
All methods share the same encoder, the same chronologically split data, and the same hyperparameter search budget, measured in trials and GPU hours. Every split is forward-in-time, training on each annotator's earlier labels and testing on their later ones. Beating a baseline means a statistically reliable gain over seeds in annotator-level macro F1 on late labels, with no loss on stationary data. On the injected-minority condition, it additionally means higher recall for the minority annotators than the global discount achieves.
Why not just…?
Why not just give each annotator their own exponential decay? This is the most serious threat, because it captures "memory depends on whose label it is" with one scalar per person. Its weaknesses are statistical. Tuning a separate $\gamma$ on a few validation labels per annotator is noisy. That is precisely the high-variance regime in which per-annotator models collapsed on contested HateXplain cases. Per-annotator decay also cannot share a population trend, so every person must relearn a shift that happened to everyone. The per-annotator-decay baseline and the fixed-$s_a$ ablation settle this directly. If tuned per-person decay matches the full model on real ordered data, we will report that the hierarchical structure is unnecessary and recommend the simpler method.
Why not just feed the timestamp to a personalized LLM? A user-ID LoRA model with a time token can, in principle, learn any person-by-time interaction. The evidence in Sensitivity, Performance, Robustness gives reason for doubt. There, models changed predictions up to 80% of the time under prompt conditioning without becoming more accurate, which suggests that conditioning tokens shift outputs without being reliably used. More importantly, interactions that are learned only implicitly give no handle for keeping a stable minority view while discarding stale labels. We test the time-token model on the injected-minority condition, where that handle is what matters.
Why not just keep a rolling window of recent labels? Windowing is cheap but throws away valid evidence. Right Now, Wrong Then argues that hard windows incur extra regret in stationary periods, compared with smooth weighting. And Unifying Data Perspectivism and Personalization finds that past exposure to a category of conflict matters, which windows discard. We include the window baseline at several values of $K$, expecting it to lose mainly on annotators who do not drift.
Why not just assume annotators are stationary? This may be correct. Most within-annotator inconsistency could be error rather than drift. That is why stage one is a measurement, why the model nests the static model as the case $s_a \to 0$, and why we inject pure label flips to check that $s_a$ stays small when the inconsistency has no temporal structure.
Decisive Checks
Three results would settle the proposal. First, on Reddit verdicts with true timestamps, does the full model beat the static history-average and per-annotator decay on later verdicts? Does a non-trivial fraction of users show a drift rate reliably above zero? Second, in the semi-synthetic condition with an early-labeling stable minority, does global discounting lose minority recall while the full model keeps it, and does the full model avoid inflating $s_a$ under pure label flips? Third, on the modified 2C NSGO, does per-source weighting keep reward accuracy for the persistent subgroup while matching NS-DPO on the migrating majority? A clear yes on the first two establishes that "who" and "when" must be modeled separately. A no on the first, with the static model matching the full model, is a publishable negative result. It would tell the perspectivist literature that frozen annotators are an acceptable simplification on current data.
References
- Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain
- ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation
- Hypernetworks for Perspectivist Adaptation
- Personalized Large Language Models
- Unifying Data Perspectivism and Personalization: An Application to Social Norms
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- The Realignment Problem: When Right becomes Wrong in LLMs
- Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions
- Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts
- Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting