RA-Rec: An Efficient ID Representation Alignment Framework for LLM-based Recommendation

Li Zhang, Xiaohan Yu, Li Zhang 0131, Xin Zhao, Yue Wang, Zhongrui MaarXiv 2024 · DOI · arXiv

Abstract

Integrating Large Language Models (LLMs) into Recommendation Systems (RS) often faces a "modality gap" between discrete user/item IDs and natural language. RA-Rec addresses this by introducing an ID Representation paradigm that aligns pre-trained ID embeddings with the LLM's latent space. By employing a lightweight, layer-specific projection module and contrastive learning, RA-Rec achieves superior recommendation performance with significantly fewer trainable parameters and less data than traditional fine-tuning methods.

Introduction

The integration of LLMs into recommendation systems typically follows two main approaches: direct ID usage or ID-to-text translation. Direct usage treats IDs as tokens, which lacks semantic depth, while translation converts IDs into descriptions, often exceeding context window limits and losing collaborative signals. RA-Rec proposes a third way: treating ID embeddings as "soft prompts" that can be understood by the LLM.

Comparison of ID integration paradigms: ID Direct usage, ID Translation, and the proposed ID Representation.

Methodology: The RA-Rec Framework

RA-Rec (Representation Alignment for Recommendation) bridges the gap between collaborative filtering and language modeling through several key components:

Hybrid Prompting and Layer-Specific Reparameterization

The framework combines "hard prompts" (textual instructions) with "soft prompts" derived from pre-trained ID embeddings. RA-Rec prepends learnable "virtual instructions" to these embeddings to guide the LLM. Furthermore, it uses a layer-specific projection module to transform ID embeddings for every transformer layer, acknowledging that different layers capture varying levels of semantic abstraction.

Detailed architecture of RA-Rec, showing the representation alignment module, user encoding process, and the dual loss functions (Contrastive and Prediction).

Alignment via Contrastive Learning

To ensure the ID embeddings are compatible with the LLM’s textual space, RA-Rec employs an InfoNCE-based contrastive loss. This aligns the transformed representations with the LLM's internal features. Crucially, the LLM backbone remains frozen, with only the alignment module being trained, making the process incredibly efficient.

Experimental Results

Evaluated on Amazon Books and Clothing datasets, RA-Rec demonstrates significant advantages:

  • Accuracy: Achieves up to a 3.0% absolute improvement in HitRate@100 over state-of-the-art baselines.
  • Efficiency: Uses approximately 40,000x fewer parameters than full fine-tuning and converges in 50% less time.
  • Data Quality: Training on a curated 6% "Efficient" subset outperformed training on the full dataset, highlighting the importance of data denoising and diversity in alignment tasks.

Conclusion

RA-Rec provides a scalable and efficient solution for leveraging LLMs in recommendation. By aligning behavioral ID embeddings with textual reasoning capabilities, it preserves the strengths of both worlds while drastically reducing the computational overhead of model adaptation.

Summary updated

BibTeX
@misc{zhang2024ra,
  author = {Li Zhang and Xiaohan Yu and Li Zhang and Xin Zhao and Yue Wang and Zhongrui Ma},
  title = {{RA-Rec: An Efficient ID Representation Alignment Framework for LLM-based Recommendation}},
  year = {2024},
  eprint = {2402.04527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.04527},
}
Figure