LightLM: A Lightweight Deep and Narrow Language Model for Generative Recommendation

Kai Mei, Yongfeng ZhangarXiv 2023 · OpenReview

TL;DR

This paper introduces LightLM, a novel, lightweight Transformer architecture specifically designed for generative recommendation. By employing a "deep and narrow" design and introducing innovative collaborative indexing methods, LightLM achieves state-of-the-art recommendation accuracy and efficiency while significantly reducing computational costs compared to existing large language models.

What Problem Does This Paper Address?

Generative recommendation, which aims to directly generate recommended items rather than scoring them individually, is a promising but challenging area. Existing approaches often repurpose heavy, general-purpose NLP Transformer models (like T5, GPT, LLaMA) for this task. However, these models are not optimized for recommendation's unique demands: they are computationally expensive, and their "wide" architecture, designed for diverse natural language, can be overkill for the relatively short and specific inputs in recommendation. Furthermore, a significant problem in generative recommendation is "hallucination," where models generate non-existent or spurious item IDs, degrading recommendation precision.

Methods

LightLM tackles these challenges with three key innovations:

  1. Deep and Narrow Architecture: Instead of the standard "deep and wide" Transformer blocks (where feed-forward layers are much wider than attention layers), LightLM uses a "deep and narrow" design. This means the inner dimension of the feed-forward layers is reduced, making the model more computationally efficient without sacrificing depth. This is motivated by the observation that recommendation inputs are typically short, requiring less representational capacity in those specific layers.
  2. Collaborative ID Indexing (SCI & GCI): To effectively capture user-item collaborative signals, LightLM introduces two novel indexing methods:
  3. Spectral Collaborative Indexing (SCI): Leverages spectral clustering on user-only, item-only, and user-item graphs to group similar users/items.
  4. Graph Collaborative Indexing (GCI): Uses Graph Convolutional Networks (GCNs) to generate embeddings for users and items, which are then quantized into collaborative IDs. This method also incorporates a hierarchical ID construction process. These methods are designed to provide richer collaborative signals than simple random indexing.
  5. Constrained Generation: To combat the hallucination problem, LightLM employs a constrained generation strategy. Instead of generating from the entire vocabulary, it uses a Trie data structure built from the valid collaborative IDs. This Trie acts as a constraint during beam search, pruning the search space and ensuring that only valid, existing item IDs are generated. This effectively merges the generation and filtering steps into a single, more efficient process.

Datasets

The experiments were conducted on three publicly available datasets:

  • Beauty and Toys: Subsets from the Amazon dataset for product recommendations.
  • Yelp: A dataset of user ratings and reviews for business recommendations.

These datasets were split using a leave-one-out strategy, where the last item in a user's history was reserved for testing, the second-to-last for validation, and the rest for training.

Key Insights and Ideas

  • "Deep and Narrow" is Sufficient: The paper convincingly argues that the "wide" architecture of standard NLP Transformers is not always necessary for recommendation tasks. Reducing the width of feed-forward layers can lead to significant efficiency gains without compromising accuracy, and in some cases, even improving it.
  • Collaborative IDs Matter: The proposed SCI and GCI methods for generating collaborative IDs are crucial. They effectively inject collaborative signals into the model, outperforming simpler indexing strategies and enabling the deep and narrow architecture to compete with larger models.
  • Constrained Generation Solves Hallucination: Directly addressing the hallucination problem with a Trie-based constraint is a clever way to ensure generated recommendations are valid, improving precision and eliminating the need for a separate filtering step.
  • Efficiency Gains are Substantial: LightLM not only matches but often surpasses existing models in accuracy while being dramatically more efficient in terms of training time, inference speed, and GPU usage.

Key Results

  • Accuracy: LightLM consistently outperformed competitive baselines (including discriminative models like BPRMF, LightGCN, SimpleX, and generative models like P5) across all three datasets. For instance, on the Beauty dataset, LIGHTLM-GCI(CoUI) achieved the highest recommendation accuracy.
  • Efficiency:
    • LightLM required significantly fewer training epochs (e.g., 8 epochs compared to 20 for P5).
  • The overall runtime of LightLM was substantially lower than P5 (e.g., 43.5 minutes vs. 1069.4 minutes).
  • GPU usage was drastically reduced, with LightLM often running on a single GPU while P5 required multiple, using significantly less memory.
  • Indexing Performance: Different indexing strategies (SCI vs. GCI, and user-only, item-only, user-item, user-item-coindexing) showed varying performance, with user-item coindexing often yielding the best results, highlighting the importance of capturing both user and item collaborative signals.

Limitations and Future Work

  • Cold-Start Problem: The paper acknowledges that LightLM's reliance on collaborative IDs makes it less effective in addressing the cold-start problem (recommending for new users or items with no interaction history).
  • Future Work: The authors suggest future research directions including:
  • Developing methods to mitigate the cold-start problem.
  • Investigating specialized Transformer architectures tailored for recommendation, such as recommendation-focused attention mechanisms.
  • Further exploring the "deep and narrow" design and its implications for various recommendation tasks.

Summary updated

Abstract

This paper presents LightLM, a lightweight Transformer-based language model for generative recommendation. While Transformer-based generative modeling has gained importance in various AI sub-fields such as NLP and vision, generative recommendation is still in its infancy due to its unique demand on personalized generative modeling. Existing works on generative recommendation often use NLP-oriented Transformer architectures such as T5, GPT, LLaMA and M6, which are heavy-weight and are not specifically designed for recommendation tasks. LightLM tackles the issue by introducing a light-weight deep and narrow Transformer architecture, which is specifically tailored for direct generation of recommendation items. This structure is especially apt for straightforward generative recommendation and stems from the observation that language model does not have to be too wide for this task, as the input predominantly consists of short tokens that are well-suited for the model's capacity. We also show that our devised user and item ID indexing methods, i.e., Spectral Collaborative Indexing (SCI) and Graph Collaborative Indexing (GCI), enables the deep and narrow Transformer architecture to outperform large-scale language models for recommendation. Besides, to address the hallucination problem of generating items as output, we propose the constrained generation process for generative recommenders. Experiments on real-world datasets show that LightLM outperforms various competitive baselines in terms of both recommendation accuracy and efficiency. The code can be found at https://github.com/dongyuanjushi/LightLM.
BibTeX
@misc{mei2023lightlm,
  author = {Kai Mei and Yongfeng Zhang},
  title = {{LightLM: A Lightweight Deep and Narrow Language Model for Generative Recommendation}},
  year = {2023},
  eprint = {2310.17488},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.17488},
}
Figure