Large Language Models as Data Augmenters for Cold-Start Item Recommendation

Jianling Wang, Haokai Lu, James Caverlee, Ed H. Chi, Minmin Chen

TL;DR

This research proposes leveraging Large Language Models (LLMs) as "offline teachers" to address the cold-start problem in recommendation systems. By generating synthetic user preferences for new items during the training phase, the framework significantly enhances the performance of efficient traditional models without incurring the high latency costs of real-time LLM inference.

The Challenge: Cold-Start vs. Real-Time Performance

Recommendation systems typically face a trade-off between accuracy and speed. Traditional models, such as Matrix Factorization, rely on "ID embeddings"—unique mathematical representations for every user and item. While effective when interaction data is plentiful, these models fail for cold-start items: new products with zero historical interactions.

Conversely, LLMs possess sophisticated reasoning capabilities and can infer an item's potential appeal from its textual description. However, LLMs are computationally "heavy." In production environments requiring sub-100ms response times across millions of items, querying an LLM for every request is infeasible. This paper explores a middle ground: harvesting LLM intelligence to pre-train traditional models before items go live.

Methodology: LLMs as Data Augmenters

The proposed framework utilizes LLMs as Data Augmenters in a three-step offline process:

  1. Pairwise Prompting: Rather than asking an LLM for an absolute preference score (pointwise), which is difficult to calibrate, the researchers use pairwise comparisons. The LLM is asked: "Given this user's history, would they prefer Item A or Item B?" where both are cold-start items.
  2. Synthetic Signal Generation: A model from the PaLM 2 family analyzes text descriptions of a user's purchase history and the new items to predict preferences.
  3. Auxiliary Training Loss: These synthetic preferences are integrated into standard recommendation backbones (e.g., NeuMF or SASRec) using an auxiliary Bayesian Personalized Ranking (BPR) loss. This guides the model to adjust embeddings based on LLM-derived insights before real user data is available.
Example of a pairwise prompt used to generate synthetic signals from the LLM.

Crucially, this augmentation occurs entirely offline during training. Once deployed, the system maintains its high-speed performance as the LLM is no longer required for serving.

Experimental Setup

The study evaluated the framework using two public Amazon datasets:

  • Amazon Beauty: Cosmetics and personal care products.
  • Amazon Sports and Outdoors: Sporting goods and equipment.

To simulate real-world conditions, the authors employed a "single-time point split," treating items appearing only in the testing period as cold-start items.

Key Insights

  • Pairwise Reliability: LLMs are more consistent at comparing two items than assigning absolute scores, mirroring human decision-making.
  • The "Teacher" Effect: While even small LLMs help, larger models (e.g., PaLM2-L) provide higher-quality synthetic data, leading to superior downstream performance.
  • Model Agnosticism: The method improved both standard matrix factorization (NeuMF) and sequential models (SASRec).
  • Bridging Content and Collaborative Filtering: This approach injects collaborative-style reasoning into models before actual interaction data exists, overcoming the limitations of pure content-based methods.

Results and Performance

The impact on cold-start recommendation was substantial. As shown in the performance table below, the LLM-augmented models outperformed baselines by orders of magnitude.

Performance comparison of NeuMF and SASRec with and without LLM augmentation across different datasets and scenarios.

On the Amazon Beauty dataset:

  • NeuMF: Recall@50 for cold-start items rose from 0.48% (baseline) to 10.53% with augmentation.
  • SASRec: Recall@50 reached 11.40%, compared to just 0.74% without augmentation.

Importantly, these gains did not significantly compromise performance for existing (warm-start) items, demonstrating the robustness of the approach.

Limitations and Future Directions

  • Cold-Start Users: The current focus is on new items. Addressing brand-new users with no history remains an open challenge.
  • Augmentation Plateau: Performance gains tend to plateau after augmenting approximately 40% of user queries, suggesting a limit to the "knowledge" extractable for a given dataset.
  • Generation Cost: While real-time serving is fast, the initial generation of millions of synthetic pairs during training prep involves significant computational costs.

Summary updated

BibTeX
@inproceedings{wang2024large,
  author = {Jianling Wang and Haokai Lu and James Caverlee and Ed H. Chi and Minmin Chen},
  title = {{Large Language Models as Data Augmenters for Cold-Start Item Recommendation}},
  booktitle = {The Web Conference},
  year = {2024},
  url = {https://openreview.net/forum?id=9kVmCj6MpB},
}
Figure