Introduction
Sequential recommendation systems aim to predict a user's next interaction based on their historical behavior. Traditionally, these systems rely heavily on Item IDs. While effective in closed environments, ID-based models face significant hurdles: they cannot handle "cold-start" scenarios (new items), they require massive embedding matrices for large catalogs, and they lack transferability across different domains.
To address these issues, researchers have turned to Text-based Sequential Recommendation (TextSRec), which utilizes text embeddings from item descriptions (e.g., from BERT). However, pre-trained language models often suffer from representation degeneration. Their output embeddings are typically "anisotropic," meaning they occupy a narrow cone in the vector space. This lack of isotropy makes it difficult for recommendation models to distinguish between items with similar but distinct semantic features.
The DWSRec Framework
This paper introduces DWSRec (Dual-View Whitening for Sequential Recommendation), a method designed to mitigate anisotropy by applying whitening transformations to text embeddings. The core idea is to transform the embedding space into a uniform, isotropic distribution, thereby enhancing item discriminability.
Dual-View Architecture
DWSRec employs a dual-view strategy to balance item discrimination with the preservation of semantic relationships:
- Dual-View Item Encoder: The model generates two distinct representations for each item. One is a fully whitened view (maximum decorrelation) to maximize uniformity. The other is a relaxed whitened view, where dimensions are grouped and whitened only within those groups to retain original semantic structures.
- Dual-View Sequence Encoder: This component uses a modified Transformer with decoupled attention. It processes the two views such that one view can guide the attention mechanism of the other, acting as a form of cross-view data augmentation.
- Dual-View Fusion: Finally, an adaptive fusion layer merges the two views using learnable attention weights to produce a robust representation for next-item prediction.
Key Insights: The Uniformity-Alignment Trade-off
A critical finding of this research is the trade-off between uniformity (how spread out embeddings are) and alignment (how well similar items stay close together). While full whitening achieves perfect uniformity, it can disrupt the semantic manifold.
By adjusting the number of "whitening groups," the authors demonstrate that a "relaxed" approach can optimize performance. As shown in empirical tests, increasing the number of groups (reducing the intensity of whitening) shifts the distribution of cosine similarities and impacts recommendation metrics like Recall@50. The dual-view approach allows the model to benefit from both a highly uniform space and a semantically rich one.
Experimental Results
The model was evaluated on three categories from the Amazon review dataset: Arts, Toys, and Tools.
- State-of-the-Art Performance: DWSRec outperformed various baselines, including ID-based models (SASRec) and specialized text-based models (UniSRec, S³-Rec). On the "Arts" dataset, it improved Recall@50 by approximately 23% over standard ID-based SASRec.
- Cold-Start Dominance: Because DWSRec relies on content rather than IDs, it showed a massive performance lead in cold-start scenarios where items were not seen during training.
- Efficiency and Convergence: Theoretically and empirically, whitening improves the condition number of the embedding matrix, leading to faster convergence and more stable training compared to using raw text embeddings.
Conclusion
DWSRec demonstrates that anisotropy in pre-trained text embeddings is a primary bottleneck for text-based recommendation. By introducing a dual-view whitening mechanism, the framework successfully balances the need for uniform item representation with the necessity of preserving semantic context. This approach provides a powerful, ID-free alternative for sequential recommendation that excels in both general and cold-start settings.
Summary updated