Sequential recommenders predict a user's next item from the ordered list of items they touched before. On real catalogs most users have only two or three interactions and most items appear a handful of times, and this sparsity is the main reason such models underperform. The standard response is to manufacture more signal from the data. Methods crop, mask, or reorder histories, run the same history twice with dropout, or add noise at inference. This proposal tests a different explanation for part of the problem. Rare items receive very few gradient updates, so their learned vectors (embeddings) may settle early near an uninformative location, typically close to the popular-item direction, and then stop moving. The model has not run out of information about these items. The optimizer has stopped exploring for them. If so, part of the sparsity penalty is premature convergence, a failure the optimization literature knows well and treats with restarts and repulsion rather than with more data.
The proposal builds and tests a small training-time intervention for recommenders that use only item IDs. It tracks, for each item embedding, whether the vector is still moving and whether that item is still predicted badly. When a rare item's vector has stagnated near the popular cluster while its loss stays high, it is partially reset: shrunk, perturbed, and given fresh optimizer state. The whole-network reset rule is borrowed from population-based training, and the distance gate is borrowed from repulsive particle swarms. Everything else keeps training as usual. The intervention is designed to compose with augmentation methods, not replace them. The scientific question it answers is whether the sparsity gap is partly an optimization trap, and the experiments are arranged so that a negative answer is as informative as a positive one.
Core Claim
The claim has two parts. The first is diagnostic. In an ID-only sequential recommender trained with Adam on sparse data, the embeddings of low-frequency items stop drifting early in training. Those stalled rows sit closer to the mean of the popular items than their eventual "useful" positions would. Users whose short histories consist of such items account for much of the model's error. The second part is the method. Resetting only those stalled rows mid-training, with a gate that fires only when a row is both stagnant and collapsed toward the popular cluster, improves accuracy on short-history users and tail items. It does so more than equal-budget alternatives: frequency-aware learning rates, whole-network restarts, or longer training. On dense data it leaves results unchanged.
What would be new is the unit and trigger of re-exploration. Existing restart methods act on whole networks across a population. Existing sparsity methods act on the data or the loss. Nobody in the file operates on the rows of the embedding table that the sparsity structure starves of gradient.
Positioning Against Prior Work
The sparsity side supplies the regime and the baselines. CL4SRec adds a contrastive loss over cropped, masked, and reordered views of each history. CT4Rec instead makes two dropout passes over the same history agree, via KL losses on outputs and batch similarity structure. The authors of Exploring and Tailoring Test-Time Augmentation perturb or delete items only at inference. All three change what the model sees and leave the optimizer untouched. CT4Rec also shows that stronger perturbation is not a free fix, since performance falls sharply past a dropout rate of about 0.8. That result constrains us: perturbations must be targeted, not global. The TTA paper also motivates us. It gains up to 38.55% on SASRec from inference-time noise, which suggests the trained model sits in a suboptimal state that small perturbations can partly undo. Its authors note, however, that it cannot change the learned parameters.
NCL-SR is the closest crossover work. It explicitly fights representation collapse under a harsh 2:2:6 split, but it builds its perturbations from a text-derived item synonym dictionary, so it does not apply to ID-only catalogs. It also finds that alignment matters more than uniformity under sparsity. That finding warns against the obvious move of adding a global "spread everything out" loss, and it is why our gate acts locally.
The premature-convergence side supplies the method. IPBT detects stalled progress in a training population and restarts members by shrinking weights and injecting noise. We reuse its shrink-perturb step but move it from whole networks to single embedding rows, and replace its population-level Gaussian-process trigger with a per-row statistic. Divergence-guided PSO adds a repulsive force that activates only when a particle's best position is close to the swarm's best. We borrow the idea of a distance gate, applied to proximity to the popular-item centroid. We also inherit DPSO's documented weakness: repulsion hurts on unimodal landscapes. This predicts our method should do nothing or harm on dense data, which we test directly. MF-PBT documents how greedy selection clones early winners. That finding motivates us to treat early "winners" in embedding space, meaning popular directions, as an attractor. INSD shows sequential networks settling into simple attractors. We considered its deflation approach and set it aside. It requires training models one after another, and the recommender's output layer scores every item, so the "readout null space" INSD relies on is essentially empty when the embedding dimension is far below catalog size.
Research Program
The work has three stages. The first measures whether the trap exists. We train SASRec (Kang and McAuley) and GRU4Rec with Adam on the benchmarks below. We log, for every item row, how often it receives a gradient, its cumulative update norm over sliding windows, and its cosine similarity to the frequency-weighted centroid of the top 10% most popular items. The trap hypothesis predicts three things. Tail rows' update norms should fall to near zero well before validation loss plateaus. Stalled tail rows should be more centroid-aligned than tail rows that keep moving. Test error on users whose histories contain stalled items should be higher even after controlling for history length.
The second stage runs an oracle test that can falsify the premise cheaply. We take a converged model and re-initialize only the stalled tail rows, keeping everything else. We then fine-tune briefly, and repeat with several random re-initializations. If the best of these does not beat the original on tail users, the stalled rows are not in a worse basin but at the only place the data supports, and the method should be abandoned.
The third stage is the method. Let $e_i$ be the embedding of item $i$ and $\Delta_i$ its cumulative update norm over the last $W$ steps in which it received gradient. Let $\ell_i$ be the running average loss when $i$ is the target and $\bar{\ell}$ that average over all items. Let $c$ be the popular-item centroid. Row $i$ is reset when three conditions hold at once. Its $\Delta_i$ must fall below a threshold $\delta$, its loss must exceed the average ($\ell_i > \bar{\ell}$), and its similarity $\cos(e_i, c)$ must exceed a gate $\gamma$. The reset is
$$ e_i \leftarrow \lambda\, e_i + (1-\lambda)\,\xi, \qquad \xi \sim \mathcal{N}(0, s^2 I), $$
where $\lambda \in (0,1)$ is the shrink factor and $s$ matches the initialization scale. Adam's moment estimates for that row are zeroed so the optimizer does not immediately push it back along the stale direction. Resets are capped at a fixed fraction of rows per epoch, and a row cannot be reset twice within a cooldown period. We will report sensitivity to $\delta$, $\gamma$, and $\lambda$, because the file flags hand-calibrated repulsion and threshold parameters as a known weakness of this family of methods. The target is a setting that transfers across datasets without per-dataset tuning.
Baselines
The naive reference is SASRec trained with Adam, tuned as its original paper specifies, with GRU4Rec as a second backbone. Every method gets the same hyperparameter search budget and the same epoch budget measured in gradient steps. That removes the objection that resets simply buy extra training.
The strongest existing competitor on ID-only sparse data is CT4Rec. It needs no external metadata, it reports more than a 26% relative NDCG@5 gain over SASRec on Amazon Beauty CT4Rec, and it has an online deployment. The file gives relative gains only, so no absolute CT4Rec NDCG value was found on Lacuna. We will rerun CT4Rec at its reported best dropout range of 0.6–0.7 within the shared budget. CL4SRec is the second augmentation baseline, with a reported average of +11.02% HR@5 and +8.50% NDCG@10 over SASRec CL4SRec, run under the same protocol. TMask-R test-time augmentation is included because it is free at training time. It reports +38.55% relative over SASRec on Amazon Home TTA. We report results on Home as well so this comparison is on the paper's own ground. NCL-SR is not run, because it requires text, so this proposal cannot beat it on its own terms. We say so rather than substituting a weakened version.
Beating a baseline means a gain on tail-user NDCG@10 that holds across five seeds with non-overlapping 95% intervals, without losing more than seed-level noise on head users. For the composable claim, the bar is that CT4Rec plus our resets beats CT4Rec alone by that margin.
Three ablations isolate the key ingredient. The first resets the same number of randomly chosen rows, which tests whether targeting matters. The second drops the centroid gate, which tests whether the popularity direction is the attractor. The third keeps Adam's moments during resets. A fourth optimization-side baseline, whole-network shrink-perturb on a fixed schedule in the style of IPBT, tests whether row-level granularity is needed at all.
Why not just…?
Why not just give rare items a larger learning rate, or use a sparse-aware optimizer? This is the most serious threat. If rows stall only because their effective step size is too small, frequency-scaled embedding learning rates or lazy sparse Adam would fix it with no reset logic. We include both as tuned baselines under the same budget. The distinguishing prediction is this: if the trap is a bad basin rather than slow movement, larger steps make stalled rows converge faster to the same centroid-aligned location, and only resets move them elsewhere. The stage-one diagnostics, rerun under the frequency-aware optimizer, show which case holds. If frequency-scaled learning rates match our method, we report that the simpler fix suffices.
Why not just add a uniformity or contrastive loss to push embeddings apart? A global spreading loss acts on every embedding at every step. NCL-SR finds that under severe sparsity alignment matters more than uniformity NCL-SR, which suggests blanket repulsion pushes genuinely similar tail items apart. CL4SRec is our instance of this family and is run as a baseline. The gate ablation also tests the local-versus-global question directly within our method.
Why not just train several seeds and ensemble, or restart the whole model? Ensembles re-explore everything at several times the inference cost, and whole-model restarts discard the well-learned head embeddings. The whole-network shrink-perturb baseline tests the restart version at equal training compute. We also report a two-seed SASRec ensemble as an honest ceiling on what untargeted re-exploration buys. If it matches our single model on tail users, targeting has no advantage beyond cost.
Why not just use test-time augmentation? TMask-R is cheap and strong, but its gains come from perturbing inputs to a fixed model. The TTA authors themselves place it below training-time methods, and embedding noise fails on GRU4Rec TTA. We report our method with and without TMask-R. If the gains simply add, the two address different failures. If TMask-R erases our gain, both are fixing the same stuck state and the cheaper one should win.
Proposed Experiments
The datasets are Amazon Beauty, Sports, Toys, and Home, plus Yelp, all with the standard 5-core filter and maximum length 50 described in the file. We also use the NCL-SR 2:2:6 sparse split on Beauty, Sports, and Toys as a harsher regime. MovieLens-1M serves as the dense control, where the DPSO precedent predicts no benefit or slight harm. Evaluation uses leave-one-out full ranking with HR@5/10 and NDCG@5/10. Every metric is reported separately for users with 2–3, 4–10, and more than 10 interactions, and for target items in frequency quintiles, because an aggregate number would hide exactly the effect under study. We will also report training time overhead, with a target of staying below the 15–25% per-iteration overhead that DPSO reports for its repulsion term.
Three checks are decisive. First, the oracle re-initialization of stage two must show a tail-user gain. Otherwise the trap premise is false and we publish the diagnostic as a negative result. Second, our method must beat the frequency-aware learning-rate baseline and random-row resets on tail-user NDCG@10 on at least the sparse-split datasets. Otherwise the benefit is step size or noise, not targeted escape. Third, the gain must shrink toward zero on MovieLens-1M and on head users. That dose-response pattern is what an optimization-trap account predicts and what a generic regularization effect would not. If all three hold, the result reframes part of the sparsity problem in sequential recommendation as an optimizer failure with a cheap, composable fix. If the first fails, it tells the field that augmentation work is not leaving optimization gains on the table.
References
- Contrastive Learning for Sequential Recommendation
- CT4Rec: Simple yet Effective Consistency Training for Sequential Recommendation
- Exploring and Tailoring the Test-Time Augmentation for Sequential Recommendation
- A Non-Contrastive Learning Framework for Sequential Recommendation with Preference-Preserving Profile Generation
- Iterated Population Based Training with Task-Agnostic Restarts
- Divergence-Guided Particle Swarm Optimization
- Multiple-Frequencies Population-Based Training
- Discovering alternative solutions beyond the simplicity bias in recurrent neural networks
- Self-Attentive Sequential Recommendation