Introduction
Sequential recommendation systems aim to predict the next item a user will interact with based on their historical sequence of actions. This task is analogous to next-token prediction in Large Language Models (LLMs). However, while LLMs typically operate on vocabularies of approximately 30,000 tokens, recommender systems must often navigate item catalogues containing millions of entries. To maintain computational efficiency at this scale, practitioners rely on negative sampling—training the model to distinguish a single "positive" interaction from a small subset of randomly selected "negative" items rather than calculating probabilities across the entire catalogue.
The Overconfidence Problem
While negative sampling enables scalability, it introduces a significant bias: overconfidence. When models like SASRec are trained using standard Binary Cross-Entropy (BCE) loss with negative sampling, they tend to predict interaction probabilities that are far higher than reality.
As shown in the figure above, a standard model may predict a near-100% probability of interaction for dozens of different items simultaneously (indicated by the high probability plateau for the top-ranked items). Since a user can realistically only interact with one item at a time, these probabilities are mathematically inconsistent. This overconfidence leads to training instability and a significant performance gap compared to "heavy" models like BERT4Rec that do not rely on the same sampling constraints.
Methodology: Generalized Binary Cross-Entropy (gBCE)
To address this, the authors introduce gSASRec, which utilizes a novel Generalized Binary Cross-Entropy (gBCE) loss function. The core innovation involves modifying the standard BCE loss by raising the sigmoid of the positive item's score to a power $\beta$.
A calibration hyperparameter $t$ (ranging from 0 to 1) is introduced to control $\beta$ based on the sampling rate.
- When $t=0$, the loss is identical to standard BCE.
- When $t=1$, the model is mathematically proven to be "calibrated," meaning its predicted probabilities should align with actual interaction likelihoods.
The gSASRec model combines this gBCE loss with the popular SASRec architecture and employs multiple negative samples per positive interaction to further refine the signal.
Experimental Evaluation
The researchers evaluated gSASRec on several benchmarks, specifically highlighting:
- MovieLens-1M: A standard benchmark for movie recommendations.
- Gowalla: A large-scale dataset containing over 1 million items. This is particularly important because "heavy" models like BERT4Rec are often computationally infeasible to train on datasets of this size, making efficient negative sampling essential.
Key Results and Insights
The study yields several critical insights:
- Architecture vs. Loss: Previously, BERT4Rec's superiority over SASRec was attributed to its bidirectional Transformer architecture. This research suggests the difference was largely due to SASRec's overconfidence under BCE. With calibrated loss, the performance of both architectures is actually quite similar.
- Performance Gains: On the million-item Gowalla dataset, gSASRec outperformed the original SASRec by 41% in terms of NDCG@10. On MovieLens-1M, using gBCE improved the Recall@1 metric for BERT4Rec by 7.2% compared to standard BCE.
- The Calibration "Sweet Spot": While $t=1.0$ is theoretically ideal for calibration, empirical results suggest $t=0.75$ provides the best balance between recommendation effectiveness and training convergence speed.
- Efficiency: The authors recommend using 128 to 256 negative samples, which provides a significant boost in accuracy without overwhelming computational resources or memory constraints.
Conclusion
gSASRec demonstrates that calibrating interaction probabilities through gBCE loss can bridge the performance gap between efficient sampled models and heavy full-catalogue models. This approach not only stabilizes training but also offers a scalable solution for recommendation tasks involving millions of items. The authors suggest that this "overconfidence" issue is pervasive in large-scale multi-class classification tasks beyond recommendation, such as document retrieval, making gBCE a potentially versatile "drop-in" improvement for the broader AI community.
Summary updated