Reembedding and Reweighting are Needed for Tail Item Sequential Recommendation

Zihao Li, Yakun Chen, Tong Zhang, Xianzhi WangWWW 2025 · OpenReview

TL;DR

This paper tackles the problem of poor performance of large language and vision models in recommending "tail items" (less popular items) in sequential recommendation. They introduce R²REC, a method that reinitializes tail item embeddings and adaptively reweights the loss function to focus more on these less popular items, significantly improving recommendation accuracy for them.

What Problem Does This Paper Address?

Modern recommendation systems increasingly leverage powerful large language models (LLMs) and large vision models (LVMs) to understand item content and user preferences. While these models excel at general recommendation tasks, they struggle with "tail items" – those with very few interactions. This is a significant issue because tail items often represent a large portion of a catalog and are crucial for diversity and user satisfaction. The paper identifies two main culprits:

  1. "All-in ground-truth" loss: The standard cross-entropy (CE) loss treats all non-target items equally, failing to provide specific optimization signals for tail items. This means tail items, by definition having fewer interactions, receive even less attention during training.
  2. "Knowledge transfer tax": The rich, pre-trained knowledge from LLMs/LVMs can dominate the optimization process, especially for tail items where historical data is scarce. This prior knowledge can inadvertently lead the model away from learning specific user preferences for these less-represented items.

Methods

The authors propose a novel approach called R²REC (Reweighting and Reembedding) to address these challenges. It consists of two core components:

  • Reembedding: For tail items, their initial embeddings (derived from LLMs/LVMs) are reinitialized using a standard Gaussian distribution. This aims to mitigate the "knowledge transfer tax" by providing a fresh, less biased starting point for these items, allowing the model to learn their representations from scratch based on user interaction data.
  • Reweighting Function: A custom reweighting function is integrated into the standard cross-entropy loss. This function adaptively adjusts the weights of items during training. It assigns higher weights to tail items and lower weights to head items, effectively encouraging the model to pay more attention to and optimize for tail item recommendations. This reweighting mechanism is mathematically shown to be comparable to Direct Preference Optimization (DPO) in LLMs, suggesting a principled way to align model behavior with desired preferences.

Datasets

The experiments were conducted on three real-world datasets:

  • Amazon Beauty
  • Amazon Toys
  • Amazon Sports

These datasets are subcategories of the Amazon reviews dataset, known for providing rich item information (text, images) and user interaction data. The authors used a standard leave-one-out strategy for splitting data into training, validation, and testing sets, with a maximum sequence length of 10. Tail items were curated based on the Pareto principle, defining the bottom 20% of items by interaction count as tail items.

Key Insights and Ideas

  • Tail items are a distinct problem for LLMs/LVMs: While these powerful models improve overall recommendation performance, they exacerbate the tail item problem due to how their knowledge is transferred and how standard loss functions operate.
  • Reinitialization is key for tail item embeddings: Simply using pre-trained embeddings for tail items can be detrimental. Reinitializing them with a simple Gaussian distribution allows the model to learn more accurate representations from the limited interaction data.
  • Adaptive loss weighting is crucial: The "all-in ground-truth" nature of CE loss is a bottleneck. A reweighting mechanism that explicitly boosts the importance of tail items during training is essential for improving their recommendation performance.
  • R²REC offers a principled approach: The reweighting function is shown to be mathematically similar to DPO, suggesting a robust theoretical foundation for its effectiveness in aligning model behavior towards tail items.

Key Results

  • Overall Performance: R²REC achieved state-of-the-art results across all three datasets, outperforming fourteen baselines, including dedicated tail-item-oriented methods.
  • Tail Item Performance: The proposed method significantly improved tail item recommendation performance, showing a large margin of improvement compared to baselines. For instance, on Amazon Beauty, R²REC achieved HR@10 of 0.0806, a substantial increase from the next best tail-item-focused baseline.
  • Ablation Studies: Removing either the reweighting or reembedding component led to a noticeable drop in performance, highlighting the importance of both. Crucially, applying R²REC's reweighting and reembedding operations to other baselines also improved their tail item performance, demonstrating the generalizability of the approach.
  • Hyperparameter Sensitivity: The study found a trade-off between overall performance and tail item performance based on the tau hyperparameter. A moderate value (e.g., 0.5) was recommended to balance these two aspects.
  • Embedding Visualization: Visualizations showed that R²REC led to sharper and more distinct item embeddings for tail items after training, compared to models without these operations.

Limitations and Future Work

  • Limitations: The paper acknowledges that while R²REC improves tail item performance, there's still a trade-off with overall performance, suggesting that a moderate tau is needed for balance. The effectiveness of cross-domain transfer learning is also discussed as a potential enhancement, though not the primary focus of R²REC itself.
  • Future Work: The authors suggest that their reweighting function, being similar to DPO, could be further explored for more precise optimization and preference alignment. Integrating cross-domain transfer learning with R²REC could also lead to further performance gains. The generalizability of the approach to other recommendation tasks and model architectures is also an avenue for future research.

Summary updated

Abstract

Large vision models (LVMs) and large language models (LLMs) are becoming cutting-edge for sequential recommendation, given their success in broad applications. Despite their advantages over traditional approaches, these models suffer more significant performance degradation on tail items against conventional ID-based solutions, which are largely overlooked by recent research. In this paper, we substantiate the above challenges as (1) all-in ground-truth, i.e., the standard cross-entropy (CE) loss focuses solely on the target items while treating all non-ground-truth equally, causing insufficient optimization for tail items, and (2) knowledge transfer tax, i.e., the knowledge encapsulated in LLMs and LVMs dominates the optimization process due to insufficient training for tail items. We propose reweighting and reembedding, a simple yet efficient method to address the above challenges. Specifically, we reinitialize tail item embedding via a Gaussian distribution to alleviate knowledge transfer tax; besides, a reweighting function is incorporated in the CE loss, which adaptively adjusts item weights during training to encourage the model to pay more attention to tail items rather than exclusively optimizing for ground-truth. Overall, our method enables a more nuanced optimization and is mathematically comparable to the direct preference optimization (DPO) in LLMs. Our extensive experiments on three public datasets show our method outperforms fourteen baselines in overall performance and improves the performance on tail items by a large margin. Our code is available at https://anonymous.4open.science/r/R2Rec-0AE0.
BibTeX
@inproceedings{li2025reembedding,
  author = {Zihao Li and Yakun Chen and Tong Zhang and Xianzhi Wang},
  title = {{Reembedding and Reweighting are Needed for Tail Item Sequential Recommendation}},
  booktitle = {The Web Conference},
  year = {2025},
  url = {https://openreview.net/forum?id=zc1XEMHbeO},
}
Figure