Abstract
Sequential Recommender Systems (SRS) excel at modeling user preferences from dense interaction data but often falter when faced with the "long-tail"—rarely purchased items and infrequent users. This paper presents LLM-ESR, a framework that leverages the semantic richness of Large Language Models (LLMs) to enhance recommendation performance in data-sparse scenarios. By integrating frozen LLM embeddings via a dual-view architecture and employing retrieval-augmented self-distillation, LLM-ESR provides a model-agnostic solution that maintains industrial-level inference efficiency.
1. Introduction
Modern recommendation engines are the invisible hands guiding digital choices. However, traditional collaborative filtering methods rely heavily on dense interaction histories. In real-world datasets, interactions typically follow a long-tail distribution, leaving a "quiet majority" of items and users with insufficient data for accurate modeling. Previous attempts to integrate LLMs into this space have struggled with computational overhead and "semantic erosion"—the loss of general knowledge during fine-tuning on specific recommendation tasks.
2. The LLM-ESR Framework
The proposed LLM-ESR framework addresses these challenges through two primary mechanisms: Dual-view Modeling and Retrieval Augmented Self-Distillation.
2.1 Dual-view Modeling
To better represent long-tail items, LLM-ESR maintains two distinct representations:
- Collaborative View: Standard trainable embeddings (e.g., item IDs) used by backbone models like SASRec or GRU4Rec.
- Semantic View: High-dimensional embeddings derived from frozen LLMs (e.g., OpenAI's
text-embedding-ada-002) based on textual item metadata such as name, brand, and price.
A trainable adapter and cross-attention mechanism are used to fuse these views, allowing the model to leverage "common sense" semantic knowledge without overfitting to sparse interaction data. To further align the views, collaborative embeddings are initialized using a dimension-reduced version (via PCA) of the LLM embeddings.
2.2 Retrieval Augmented Self-Distillation
For users with sparse interaction histories, LLM-ESR identifies "semantic neighbors"—users with similar preference profiles as determined by LLM-generated embeddings from historical interactions. The model then uses Self-Distillation, treating the aggregated preferences of these neighbors as a "teacher" signal to refine the representation of the long-tail "student" user.
3. Experimental Setup
The researchers evaluated LLM-ESR using three diverse datasets: Yelp (business reviews), Amazon Fashion, and Amazon Beauty. The framework was tested across three popular backbone models: GRU4Rec, Bert4Rec, and SASRec. Crucially, all LLM embeddings were cached, ensuring that the system adds zero LLM inference overhead during deployment.
4. Key Results and Analysis
The experimental results demonstrate that LLM-ESR consistently outperforms state-of-the-art baselines across all metrics, including Hit Rate (H@10) and NDCG (N@10).
4.1 Performance in the Long-tail
The most significant gains were observed in the "Tail Item" and "Tail User" segments. For instance, on the Yelp dataset with a SASRec backbone, the improvement for tail items was significantly higher than for head items. This confirms that the semantic signals from the LLM effectively compensate for the lack of collaborative data. Ablation studies further confirmed that both the semantic view and the self-distillation components are critical for these improvements.
4.2 Efficiency and Embedding Quality
By keeping LLM embeddings frozen and using PCA for initialization, the model preserves rich semantic relationships while accelerating convergence. Analysis of the embedding space suggested that LLM-ESR produces a more uniform distribution, reducing the popularity bias common in traditional recommenders and creating a healthier recommendation space.
5. Conclusion
LLM-ESR represents a practical advancement in sequential recommendation. By decoupling semantic understanding from collaborative learning and caching LLM outputs, it provides the benefits of deep language modeling without the associated latency costs. While the framework introduces new hyperparameters like distillation weight, its model-agnostic nature makes it a versatile tool for improving recommendations for the "quiet majority" in industrial applications.
Summary updated