Two systems that learn about a person over months or years have the same weakness. Speaker verification, which checks whether a voice matches an enrolled template, and preference-tuned language models, which learn what a user wants from pairwise feedback, both treat old data as if it were as current as new data. Voices change as people age, and so do tastes. Each field has found this independently. Speaker verification error on a three-year longitudinal corpus rose 78% relative for male speakers (Evaluation and calibration of short-term aging effects in speaker verification). Standard preference optimization falls to near-chance accuracy when preferences flip (Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift). Each field also fixes the problem the same crude way: one global rule for how quickly old evidence should lose weight, set by hand or tuned on validation data.
This proposal argues that the right forgetting rate is a property of the individual and can be estimated from that person's own history. The idea comes from speaker-aging research, which separates directional change (true drift) from non-directional scatter (session noise). If each user's history is modeled as a slowly wandering true state observed through noise, the ratio of wander to noise for that user fixes the optimal discount in closed form. The same estimator can set per-speaker template updating and per-user preference discounting. In a voice assistant it also links the two, because the estimated identity uncertainty decides how much a given utterance's feedback counts toward that user's preference profile.
Core Claim
Both literatures report that drift rates differ widely across people, but neither uses this. In speaker aging, the VoxAging study found female speakers deteriorating faster than males in English (VoxAging). A mixed-effects analysis found the gender ordering reversed in Finnish (Speaker Verification Across Ages). The MARP calibration work notes that some individual speakers drift quickly while others barely move. In preference drift, NS-DPO uses a single discount $\gamma$ for all data and admits that choosing it requires knowing an unobservable "variation budget." The claim is that a pooled, per-user estimate of the drift-to-noise ratio beats any single tuned global discount whenever users are heterogeneous, and that this holds in both domains with the same estimator.
The model is a local-level state-space model, meaning a random walk seen through noise. For user $u$, the true state $\theta_{u,t}$ is either a voice centroid in embedding space or a reward direction. It moves by process noise of variance $q_u$ per unit of real elapsed time, and each observation adds noise of variance $r_u$. For regularly spaced observations the optimal estimator is exponential smoothing, which is the standard steady-state Kalman filter result. Its per-step retention factor depends only on the ratio $s_u = q_u / r_u$:
$$ \gamma_u = 1 - \frac{p_u}{p_u + 1}, \qquad p_u = \frac{s_u + \sqrt{s_u^2 + 4 s_u}}{2}. $$
Here $\gamma_u$ plays the same role as NS-DPO's $\gamma$ but is specific to the user. The contribution is not the filter, which is textbook. It is three claims: $s_u$ can be estimated reliably from sparse per-user histories by pooling across users; plugging it in removes the variation-budget problem; and the filter's predicted uncertainty gives time-aware calibration for free. The last point matters for voices. The predicted variance of a new observation grows with the time since the last accepted session, so the verification score threshold widens with elapsed time as a derived consequence rather than a separately fitted quality function.
Research Program
The first component is the voice tracker. We take frozen embeddings from a strong speaker encoder (ECAPA-TDNN and ERes2Net-Large, the architectures VoxAging found to degrade slowest) and run a per-speaker filter on the enrollment centroid. Sessions can be irregularly spaced, so process variance scales with real elapsed time. The filter updates only with a weight equal to the calibrated probability that the session is the enrolled speaker. That probability comes from the filter's own prediction and a pooled impostor model, which guards against the drifting-impostor failure of naive template replacement. We estimate $q_u$ and $r_u$ by empirical Bayes: a shared prior across speakers, with per-speaker posteriors shrunk toward it. This follows the logic of A Shared Low-Rank Adaptation Approach to Personalized RLHF, which shares structure across users to survive per-user data scarcity. We reuse its idea at the level of drift statistics rather than weights.
The second component is the preference tracker. We use shared features from a reward model, with a per-user linear head that plays the role of P-ShareLoRA's user-specific projection. We fit a random walk on that head under a Bradley-Terry likelihood, using a Laplace approximation because the likelihood is not Gaussian. From each user's history we estimate $s_u$ and then train with the NS-DPO loss, each pair weighted by $\gamma_u^{T - t_i}$ instead of a global $\gamma^{T-t_i}$. NS-DPO supplies the training objective. Our change is only where the discount comes from.
The third component is the coupling in a simulated lifelong voice assistant. Each preference observation is attributed to user $u$ with weight equal to the voice tracker's posterior that the speaker is $u$. Without this weighting, two failures appear. Cross-age misidentification can write feedback into the wrong profile. Identity fragmentation can split one user's history into phantom profiles, the "major speaker splitting" effect reported for lifelong diarization in Towards lifelong human assisted speaker diarization. This component is deliberately modest. No benchmark in the literature pairs longitudinal voice with longitudinal preferences, so we build a semi-synthetic one: VoxAging speakers' real multi-year audio, each assigned a preference trajectory taken from real annotator data with injected change points. The coupling in this setup is synthetic, and we say so. The experiment tests the attribution mechanism, not any claim that aging voices predict aging tastes. We explicitly do not pursue the speculative idea of mining the age embedding that Disentangling Age and Identity with a Mutual Information Minimization discards for preference signal, because no available data could test it.
Baselines
On the voice side, the naive reference is static enrollment from the first sessions with a fixed threshold. For VoxAging, the paper's reported degradation from an initial 0.57% EER (equal error rate, where false accepts equal false rejects) for ERes2Net-Large on Vox-O to a 10-year female EER of 6.77% gives the scale of the problem.
The strongest existing competitors are three methods, run together and in combination. Temporally spaced enrollment, which builds templates from the first, middle and last sessions, was reported to improve over baseline on RedDots (Exploring Session Variability and Template Aging); no numeric value is reported on Lacuna for it. Elapsed-time calibration with quality measure functions reduced miscalibration by 37% at 16–19 month gaps on MARP, where EER rose from 2.44% to 4.35% for male speakers. The AA-MIM age-disentangled encoder reported 1.53% relative EER and 4.79% relative minDCF gains over ResNet34+ADAL on Vox-CA.
We also include rolling template updating (replace or average with each accepted session) and a sliding window of the last $k$ sessions, both with a global tuned rate. The key ablation is our filter with a single global $s$ tuned on development speakers. This isolates whether per-user estimation, rather than filtering itself, drives any gain. All methods use identical frozen embeddings and the same chronological train/development/test speaker split. Every tunable rate or window gets the same grid budget on development speakers. Beating the baselines means lower EER and lower calibration loss on the final third of each test speaker's timeline, and no increase in impostor acceptance under a slow-impostor attack in which a different speaker's sessions are gradually substituted.
On the preference side, the naive reference is standard DPO on all history, which the NS-DPO paper reports collapsing to about 50–55% reward accuracy under flips. The strongest competitor is NS-DPO with a global $\gamma$ tuned by grid search on held-out users. The paper reports 60–70% reward accuracy under drift, more than 10 points over baselines in gradual-drift scenarios, and a length-controlled win rate of 8.93 versus DPO's 7.29 on UltraFeedback-LM. We add P-ShareLoRA without discounting, reported at 74.65% accuracy on Reddit TL;DR with Llama-3 8B in the stationary setting, and CIPHER-style retrieval of induced preference descriptions (Aligning LLM Agents by Learning Latent Preference from User Edits). All run on the same backbone with the same number of preference pairs and gradient steps, and the same tuning budget spent on $\gamma$, windows or retrieval $k$. The ablation is again per-user $\gamma_u$ versus the best global $\gamma$. Beating the baselines means higher reward accuracy on each user's current preferences, averaged over users so that slow-drifting users are not swamped by fast ones. No reported result was found on Lacuna for any method on per-user heterogeneous drift, so the comparison numbers there will be our own.
Why not just…?
Why not just tune one global discount on validation data? This is the real threat. If users drift at similar rates, a well-tuned global $\gamma$ is optimal and our method only adds estimation noise. The evidence for heterogeneity is strong on the voice side: gender and language interactions reverse between English and Finnish, and the MARP work points to idiosyncratic per-speaker trajectories. On the preference side, heterogeneity is plausible but unmeasured. The per-user versus global ablation settles this directly. We will also report the estimated spread of $s_u$ across users. If that spread is small, we will report that per-user rates are unnecessary in that domain rather than claim a win.
Why not just remove age from the embedding so there is nothing to track? Disentanglement such as AA-MIM attacks the voice problem at the representation level. But the mixed-effects study still finds negative age-mismatch effects in modern ECAPA-TDNN embeddings, the disentanglement losses are hard to balance without stripping identity cues, and the approach says nothing about preferences. We run our tracker on top of AA-MIM embeddings. If tracking adds nothing there, disentanglement suffices for voices and the voice half of our claim fails.
Why not just detect change points and reset? A reset handles abrupt flips well, and the UltraFeedback flip benchmark is exactly that regime. It throws away all prior data on each detection, however, and it handles slow, continuous drift like vocal aging poorly. We include a change-point-and-reset baseline. We expect it to match us on abrupt flips and lose on gradual drift and on the multi-year voice data, and the experiments report both regimes separately.
Why not just keep the whole history in context and let a large model sort it out? CIPHER shows that induced preference descriptions beat raw history stuffing. Its authors state that it assumes preferences are stable within a context. We therefore expect retrieval over stale descriptions to inherit preference inertia, and we test CIPHER with and without $\gamma_u$-weighted retrieval.
Proposed Experiments
The first decisive check runs on VoxAging, which has 293 speakers with weekly sessions spanning up to 17 years. For every test speaker we plot EER and calibration loss against years since enrollment. We report results separately by gender, language and age bracket, with the 40–50 group of particular interest because VoxAging found its decline steepest. The prediction is that per-speaker filtering flattens the degradation curve most for the fastest-drifting speakers and matches the global filter for stationary ones. We replicate on LOCUST, whose Supreme Court speakers span about 19 years on average. Its male skew makes it a check on generalization rather than on gender effects. We also use MARP for short-term calibration, where we can compare directly to the reported 37% miscalibration reduction.
The second check runs on preferences. We build heterogeneous-drift versions of TV-HH, UltraFeedback-RM and 2C NSGO in which simulated users follow change points and gradual drift at rates drawn from a wide distribution. We also build a variant of the Reddit TL;DR five-annotator setup in which annotators' pairs are interleaved into time-ordered streams with injected per-annotator swaps. We sweep the spread of drift rates from zero upward. The key output is the crossover curve: the spread at which per-user $\gamma_u$ starts to beat the best global $\gamma$, and whether at zero spread it loses only slightly.
The third check runs on the semi-synthetic voice-assistant stream. We measure preference accuracy per true user under three attribution rules: identity from static verification, from our voice tracker with hard assignment, and from posterior-weighted soft assignment. We also count how often a user's history is split across profiles. The coupling is worth keeping only if soft attribution measurably reduces wrong-profile updates without slowing adaptation. If not, a decoupled pipeline with a good aging-aware verifier is enough, and the note's lasting contribution is the shared per-user forgetting rule.
References
- Evaluation and calibration of short-term aging effects in speaker verification
- Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
- VoxAging: Continuously Tracking Speaker Aging with a Large-Scale Longitudinal Dataset in English and Mandarin
- Speaker Verification Across Ages: Investigating Deep Speaker Embedding Sensitivity to Age Mismatch in Enrollment and Test Speech
- A Shared Low-Rank Adaptation Approach to Personalized RLHF
- Towards lifelong human assisted speaker diarization
- Disentangling Age and Identity with a Mutual Information Minimization for Cross-Age Speaker Verification
- Exploring Session Variability and Template Aging in Speaker Verification for Fixed Phrase Short Utterances
- Aligning LLM Agents by Learning Latent Preference from User Edits