Forget the Relationship, Not the Rarity: Region-Specific Forgetting for Imbalanced Regression Under Drift

Many systems predict a continuous quantity whose distribution is badly skewed and whose meaning changes over time. Examples include how long a user will watch a video, how strongly an annotator rates a response, or how high a pollution reading will go. Two literatures handle the two halves of this problem separately. Imbalanced regression reweights or regularizes training so that rare target values (the long tail of very long watch times, very low ratings) are not swallowed by the common ones. Work on preference drift down-weights old data so the model tracks what users want now. Combining them naively breaks both. Exponential forgetting removes old samples at the same rate everywhere, and tail samples are exactly the ones that cannot be replaced quickly, so the model loses its picture of the extremes and collapses toward the recent majority.

The proposal starts from a simple observation. "The data changed" covers two different things that call for opposite responses. If only the frequency of target values changed (more short watches this month), old data still describe the input-to-output relationship correctly. They should be kept, and the frequency shift corrected with an updated prior. If the relationship itself changed, old data in that part of the target range should be forgotten. We propose to estimate, separately for each region of the target axis, whether the relationship has drifted. Because the tail has too little recent data to answer that question alone, the estimate borrows strength from neighbouring regions. Each region then forgets at its own rate, while a continuously updated balancing prior absorbs changes in target frequencies. In short, the model forgets stale relationships, not rare values.

Core Claim

Under non-stationary continuous feedback with a skewed target, a single global decay rate is the wrong control knob. Its best setting trades fast adaptation in the dense bulk of the data against memory in the sparse tail, and no single value serves both. Separating marginal shift (correct it with a time-varying prior) from conditional drift (forget, but only where it happens) should improve balanced error across all target regions. The largest gains should appear in few-shot tail regions after a change in the bulk. It should do no worse than global decay when drift really is global. The claim is falsifiable. If a well-tuned global decay, combined with a balanced loss whose prior is also decayed, matches the method on real drifting data, the extra machinery is not justified.

Grounding in Prior Work

The imbalanced-regression side supplies the method components. Delving into Deep Imbalanced Regression introduced label distribution smoothing, which blurs the label histogram with a kernel so that scarcity is treated as a property of a neighbourhood rather than a single value, and feature distribution smoothing, which shares feature statistics across neighbouring label bins. We reuse the first idea in a new role. Instead of smoothing label counts, we smooth a drift statistic along the label axis so that sparse tail regions inherit evidence from adjacent ones. Balanced MSE for Imbalanced Visual Regression recasts squared error as a Gaussian likelihood and corrects it with the training label prior $p_{\text{train}}(y)$. That prior is fixed in their work. We replace it with a time-indexed prior $\hat p_T(y)$, which is the component that handles pure label shift. RankSim and PRIME regularize feature geometry across the label range. They motivate the concern that discarding tail anchors damages the representation, but both assume i.i.d. batches and say nothing about time.

The drift side motivates the problem and supplies the strongest temporal baseline. Right Now, Wrong Then (NS-DPO) weights each preference pair by $\gamma^{T-t_i-1}$ and proves regret guarantees for this discounting. Its discount is uniform across samples, and its rate depends on an assumed variation budget. We keep exponential discounting but let the rate depend on the target region and estimate it from data. Continuous-Utility DPO argues that continuous utility scores carry more information than binary comparisons, which supports our choice to regress continuous feedback rather than binarize it. It does assume a stationary utility. DRIFT shows that real feedback is heavily imbalanced (positive signals are 1–3%) but treats imbalance as a binary class ratio, not a skewed continuous target.

One paper acts as a constraint on evaluation. Imbalanced Regression Evaluation Under Uncertain Domain Preferences shows that rankings of imbalanced-regression methods flip under small perturbations of the relevance function that defines "extreme", reaching 89% rank-shift probability on one dataset. A claimed win at a single relevance setting is therefore weak evidence, and we report results across perturbed relevance functions.

Research Program

The model sees a stream of triples $(x_i, y_i, t_i)$, where $x_i$ is the input (user and context features), $y_i$ is a continuous target, and $t_i$ is a timestamp. At current time $T$, each sample's loss is

$$ \ell_i = \gamma(y_i)^{\,T-t_i}\;\mathcal{L}_{\text{bal}}\big(f(x_i), y_i;\ \hat p_T\big), $$

where $f$ is the predictor and $\mathcal{L}_{\text{bal}}$ is the Balanced MSE likelihood with its label prior replaced by $\hat p_T$. The prior $\hat p_T(y)$ is a kernel density estimate of recent labels computed with fast decay. This is feasible because estimating label frequencies needs only labels, not the slower-to-verify input–output relationship. The factor $\gamma(y) \in (0,1]$ is a region-specific retention rate.

The retention rate is set by a drift statistic. For a grid of target regions $b$, we compare the current model's residuals on recent samples in $b$ with its residuals on held-out older samples in $b$. The comparison is a standardized difference in mean squared residual, $d_b$. We then smooth it along the label axis,

$$ \tilde d(y) = \sum_b k(y, y_b)\, n_b\, d_b \Big/ \sum_b k(y, y_b)\, n_b, $$

where $k$ is a Gaussian kernel, $y_b$ is the centre of region $b$, and $n_b$ is its recent sample count. Count weighting means data-poor tail regions lean on their neighbours rather than on noise. Finally, $\gamma(y)$ is a monotone map from $\tilde d(y)$: no detected drift gives a slow floor rate, and strong drift gives a fast ceiling rate. Both rates are tuned once on validation data.

The construction has a useful fallback property. If drift is detected everywhere, the method reduces to NS-DPO-style global decay applied to a balanced loss. If nothing drifts except label frequencies, it reduces to Balanced MSE with an updated prior. The research program studies when the cases in between occur in real data and whether the drift estimate is reliable enough to act on. Feature-space regularizers such as RankSim can be added on top, but they are not part of the core claim.

Baselines

The naive reference is a regressor trained with plain MSE on all accumulated data with no forgetting and no balancing. It shows how bad tail error and post-drift error are when nothing is done.

The strongest existing combination, and the main threat, is Balanced MSE with its prior estimated from decayed counts, with every sample's loss multiplied by a single global factor $\gamma^{T-t_i}$ in the manner of NS-DPO. Because the prior is decayed too, old tail samples get a low time weight but also a low density, so the balancing partly compensates. This makes it a much stronger competitor than the naive "LDS times decay" version. We give it the same backbone, data stream, and number of training updates as our method. Its $\gamma$ is chosen by a grid search on rolling validation windows with the same tuning budget as our two rates. We also report an oracle variant that tunes $\gamma$ on the test stream, as an upper bound on any global decay.

We also run three further baselines. The first is Balanced MSE with the time-varying prior and no decay, which isolates label-shift correction. The second is static Balanced MSE, LDS+FDS, and RankSim retrained on a sliding window. The third is NS-DPO-style global decay on plain MSE.

Two ablations isolate the key ingredient. One uses region-specific decay without kernel smoothing of the drift statistic, which tests whether borrowing strength across the label axis matters. The other uses smoothed region-specific decay with a frozen prior, which tests whether separating marginal from conditional drift matters.

Published numbers exist only for the static setting. On AgeDB-DIR, RankSim reports MAE 7.77 for the vanilla model, 7.13 with RankSim, and 6.94 with RankSim+FDS. On IMDB-WIKI-DIR, Balanced MSE reports balanced MAE falling from 32.78 to 28.14. Our first check is to reproduce these figures before introducing drift. For drift, NS-DPO reports that DPO accuracy falls to roughly 50–55% after change points while NS-DPO stays above 60–70% on UltraFeedback, but that is pairwise accuracy, not regression. There is no reported result found on Lacuna for any imbalanced-regression method under temporal drift, nor for continuous watch-time regression on KuaiRand. These numbers will be established here.

Beating the strongest baseline means three things. Few-shot-region balanced MAE in the windows after a change must be lower than that of the tuned global-decay balanced baseline, with a gap larger than seed variance across at least five seeds. Many-shot error must not be worse. The advantage must hold across most perturbed relevance functions rather than at one setting.

Why not just…?

Why not just use a sliding window with Balanced MSE or LDS? A window throws away old tail samples just as hard decay does. Label smoothing over a short window has empty tail bins to smooth, which is the "missing data holes" problem noted in Delving into Deep Imbalanced Regression. NS-DPO's analysis also favours smooth discounting over hard cutoffs. We include window variants at several widths, so if a well-chosen window matches our method the experiments will show it.

Why not just multiply a balanced loss by one global decay, with the prior also decayed? This is the real threat. The density correction partly restores old tail samples, and when drift affects all target regions roughly equally, a single rate is the right model and our method should only tie. The argument for region-specific rates holds only when drift is uneven across the label axis. The semi-synthetic experiments vary exactly this, and the KuaiRand experiment tests whether uneven drift occurs naturally. If the oracle-tuned global decay matches us on KuaiRand, we will report that region-specific forgetting is not needed in practice.

Why not just bin the target and use streaming imbalanced classification? Binning discards the ordering that lets neighbouring values inform each other, which is the central argument of Delving into Deep Imbalanced Regression. Continuous-Utility DPO argues that coarse supervision costs sample efficiency, and tail regions can least afford that cost. We run a cost-sensitive binned classifier with decay as an additional baseline, but we expect it to lose on within-bin error. That is a weaker comparison than the global-decay one.

Why not just add time as an input feature and let the network adapt? Time features can model drift that recurs or is predictable, but they cannot extrapolate to a new regime. They also do nothing about the gradient imbalance that drowns the tail. We test a time-feature variant of the strongest baseline.

Proposed Experiments

The first testbed is semi-synthetic. We turn AgeDB-DIR and IMDB-WIKI-DIR into streams by ordering samples into time steps and injecting controlled changes. Pure label shift is created by resampling to change age frequencies. Bulk conditional drift is created by applying a label transform only to common ages, and tail conditional drift by applying it only to rare ages. Each scenario includes abrupt and gradual variants. These scenarios make the predictions concrete. Under pure label shift, our method should match prior-only Balanced MSE and beat any decay. Under bulk drift, it should beat global decay on tail error. Under tail drift, it should match global decay. A failure in the bulk-drift case would falsify the core claim.

The second testbed is real. KuaiRand logs timestamped interactions under randomized exposure, which reduces confounding from the recommendation policy. We regress continuous engagement such as play-time ratio, a heavy-tailed target, and evaluate prequentially (predict each day, then train on it). KuaiSim is used only as a secondary check, generating patience-driven engagement shifts.

The third testbed adapts the UltraFeedback drift protocol of NS-DPO to reward regression on continuous ratings. This tests whether the effect carries over to preference data. If the PM10 air-pollution sets used by Costa and Moniz keep their temporal order, they provide a natural extreme-value stream.

We report several metrics: balanced MAE and few-shot-region MAE per time window, recovery time after a change, and SERA (squared-error relevance area) under the convolution and elastic perturbations of the relevance function. We also report the probability of rank shift between methods, in the style of Costa and Moniz.

A diagnostic check is whether the smoothed drift statistic actually flags the injected drift region in the semi-synthetic streams. If it does not, any gain must come from somewhere else and the mechanism claim fails. The decisive check is the KuaiRand comparison against oracle-tuned global decay with a decayed balanced prior. A gain in post-change tail error that holds across relevance perturbations supports the proposal. A tie means it does not.