Are ID Embeddings Necessary? Whitening Pre-trained Text Embeddings for Effective Sequential Recommendation

Lingzi Zhang, Xin Zhou, Zhiwei Zeng, Zhiqi Shen

Abstract

Sequential recommendation traditionally relies on item ID embeddings, which suffer from cold-start issues and poor transferability. While pre-trained text embeddings offer a potential solution, they are often hindered by anisotropy—a phenomenon where embeddings cluster in a narrow cone. This paper introduces WhitenRec and WhitenRec+, which utilize whitening transformations to improve the uniformity and discriminative power of text embeddings. Our results show that these text-only models can outperform state-of-the-art hybrid models, particularly in cold-start scenarios.

Introduction

Sequential recommendation systems aim to predict future user interactions based on historical sequences. The standard approach involves learning unique ID embeddings for every item. However, this dependency creates significant challenges: high memory consumption for large item catalogs, inability to handle "cold-start" items (new items with no history), and a lack of transferability across different domains or platforms.

The Anisotropy Problem

Recent research has explored using pre-trained language models (PLMs) like BERT to extract item features from text (titles, descriptions). However, these embeddings often exhibit "anisotropy," where vectors are highly correlated and occupy a narrow subspace. This makes it difficult for recommendation models to distinguish between semantically distinct items, as their average cosine similarity remains excessively high (often > 0.8). In essence, the model perceives most items as nearly identical, limiting its predictive accuracy.

Methodology: WhitenRec and WhitenRec+

To address anisotropy, we propose two frameworks that apply whitening transformations to PLM-extracted features before they enter the sequence encoder.

Whitening Transformation

We employ Zero-phase Component Analysis (ZCA) whitening to transform the anisotropic distribution of text features into an isotropic (spherical) Gaussian distribution. This process "spreads out" the items in the vector space, enhancing the model's ability to discriminate between items while maintaining the original semantic relationships.

Relaxed Group Whitening

While full whitening improves uniformity, it may occasionally disrupt the underlying semantic manifold—the natural grouping of similar items. We introduce "group whitening" to relax the transformation, preserving local semantic clusters while still improving the global distribution of the embedding space.

Model Architectures

The proposed WhitenRec and WhitenRec+ architectures compared to standard SASRec baselines. Are ID Embeddings Necessary? Whitening Pre-trained Text Embeddings for Effective Sequential Recommendation, page 4

The proposed WhitenRec and WhitenRec+ architectures compared to standard SASRec baselines.

As shown in Figure 2, WhitenRec applies a single whitening transformation followed by a projection head. WhitenRec+ utilizes an ensemble approach, combining both fully whitened and relaxed whitened representations through a shared projection head to balance discriminative power with semantic awareness.

Experiments and Results

We evaluated our models on four benchmarks: Amazon Arts, Crafts and Sewing; Toys and Games; Tools and Home Improvement; and a Food recipe dataset. For the Amazon datasets, item titles, categories, and brands were concatenated to create the text input for the BERT encoder.

Performance and Efficiency

WhitenRec+ consistently outperformed state-of-the-art models like SASRec and UniSRec. In warm-start settings, Recall@20 improved by 10-16% over standard baselines. Notably, WhitenRec+ achieved these results with significantly fewer parameters—approximately 50% fewer than UniSRec on the Tools dataset—due to the removal of the massive ID embedding matrix.

Visualizing the Impact of Whitening

t-SNE visualizations of item embeddings: (a) raw pre-trained text embeddings, and (b-d) whitened embeddings with varying group sizes (G). Are ID Embeddings Necessary? Whitening Pre-trained Text Embeddings for Effective Sequential Recommendation, page 5

t-SNE visualizations of item embeddings: (a) raw pre-trained text embeddings, and (b-d) whitened embeddings with varying group sizes (G).

The effect of our approach is clearly visible in Figure 6. Raw BERT embeddings (a) are tightly clustered in a small region, whereas whitened embeddings (b-d) are more uniformly distributed across the space, facilitating better recommendation performance by increasing item discriminability.

Cold-Start Performance

The most significant gains were observed in cold-start scenarios, where the model must recommend items it has never seen during training. On the Food dataset, WhitenRec+ improved NDCG@50 by a staggering 64.5% over the best baseline, demonstrating the robustness of text-based features when properly conditioned via whitening.

Conclusion

This work demonstrates that ID embeddings are not strictly necessary for effective sequential recommendation. By applying whitening transformations to pre-trained text embeddings, we can overcome anisotropy and build models that are more efficient, transferable, and performant, especially in challenging cold-start environments. Future research will explore optimal fusion techniques for IDs and whitened text to further enhance recommendation quality without sacrificing uniformity.

Summary updated

BibTeX
@inproceedings{zhang2024are,
  author = {Lingzi Zhang and Xin Zhou and Zhiwei Zeng and Zhiqi Shen},
  title = {{Are ID Embeddings Necessary? Whitening Pre-trained Text Embeddings for Effective Sequential Recommendation}},
  booktitle = {ICDE},
  year = {2024},
  doi = {10.1109/icde60146.2024.00063},
}
Figure