RecGPT: Generative Pre-training for Text-based Recommendation

Hoang Ngo, Dat Quoc NguyenACL 2024

TL;DR

This paper introduces RecGPT, a novel domain-adapted large language model (LLM) specifically for text-based recommendation tasks. By pre-training on a massive recommendation-specific corpus and fine-tuning for instruction following, RecGPT-7B-Instruct significantly outperforms existing state-of-the-art models in both rating prediction and sequential recommendation.

What Problem Does This Paper Address?

Traditional recommendation systems often rely on matrix factorization or deep learning models that struggle with data sparsity, flexibility, and scalability. While recent LLMs show promise, they often treat users and items as mere IDs, failing to leverage their rich textual information. This paper addresses the gap of effectively integrating LLMs with the inherent textual nature of recommendation data (user reviews, item descriptions) to build more powerful and context-aware recommendation systems. The authors aim to create a unified model that can handle various recommendation tasks by leveraging the generative capabilities of LLMs.

Methods

The core of this work is the development of the RecGPT model series.

  • RecGPT-7B: This is the base, domain-adapted LLM. It's a Transformer decoder-based model incorporating techniques like flash attention and ALiBi for extended context handling. Crucially, it's initialized with weights from the MPT-7B model and then continually pre-trained on a massive, recommendation-specific corpus. This domain adaptation is key to its performance.
  • RecGPT-7B-Instruct: This is an instruction-following variant of RecGPT-7B. It's fine-tuned on a dataset of instructional prompts and responses specifically designed for recommendation tasks, such as rating prediction and sequential recommendation. This fine-tuning allows the model to understand and execute specific recommendation queries.

The training process utilizes the LION optimizer and sharded data parallelism with FSDP for efficient large-scale training.

Datasets

The authors leverage a comprehensive collection of publicly available datasets for training and evaluation, including:

  • Amazon Product datasets (various categories like Beauty, Sports, Toys, Books, etc.)
  • Anime
  • BookCrossing
  • Food
  • Goodreads
  • HotelRec
  • MovieLens
  • Netflix
  • Steam
  • WikiRec
  • Yelp

For pre-training, they created a massive corpus of 20.5 billion tokens from these datasets. For fine-tuning, they curated a dataset of over 100,000 instructional prompt-response pairs. A significant contribution is the careful handling of data leakage by ensuring that users in the fine-tuning set do not overlap with those in the pre-training set, and by implementing a consistent test set strategy for evaluation.

Key Insights and Ideas

  • Domain Adaptation is Crucial: Simply using a general-purpose LLM for recommendation is suboptimal. Continually pre-training an LLM on a large, recommendation-specific corpus (like their 20.5B token dataset) significantly boosts performance.
  • Textual Representation Matters: Representing items by their textual descriptions and users by their interaction histories (reviews, titles) is more effective for LLMs than using simple IDs, as it taps into the LLM's natural language understanding capabilities.
  • Instruction Following for Recommendation: Fine-tuning LLMs to follow specific instructions (e.g., "Predict the rating for the last item") makes them directly applicable to recommendation tasks, bridging the gap between general language models and specialized recommender systems.
  • Data Leakage is a Real Problem: The paper highlights a subtle data leakage issue in previous recommendation benchmarks where training data for one task might overlap with test data for another, leading to inflated performance metrics. Their proposed consistent test set strategy addresses this.

Key Results

  • State-of-the-Art Performance: RecGPT-7B-Instruct consistently outperforms strong baselines, including P5 and even ChatGPT (in a few-shot setting), on both rating prediction and sequential recommendation tasks across four benchmark datasets.
  • Rating Prediction: RecGPT-7B-Instruct achieves the best results in terms of RMSE and MAE, setting new state-of-the-art scores.
  • Sequential Recommendation: While ChatGPT shows limitations due to a lack of in-domain data, RecGPT-7B-Instruct demonstrates competitive results, notably outperforming P5 on "Sports and Outdoors" and "Toys and Games" datasets.
  • Ablation Study: The paper shows that the continual pre-training of RecGPT-7B is essential, as a baseline MPT-7B model fine-tuned for instruction following (without the recommendation-specific pre-training) performs substantially worse.

Limitations and Future Work

  • Intrinsic Memory Limitation: The LLM's knowledge is limited to its training data. It may struggle with items not seen during training, potentially leading to irrelevant information or hallucinations. This is a general limitation of LLM-based methods.
  • Task Scope: The current work focuses on rating prediction and sequential recommendation. The authors plan to explore other recommendation tasks in future work.
  • Data Leakage Mitigation: While they address data leakage for evaluation, the paper notes that previous works might have been affected by it, suggesting a need for more robust evaluation protocols in the field.

Summary updated

BibTeX
@inproceedings{ngo2024recgpt,
  author = {Hoang Ngo and Dat Quoc Nguyen},
  title = {{RecGPT: Generative Pre-training for Text-based Recommendation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure