Enhancing High-order Interaction Awareness in LLM-based Recommender Model

Xinfeng Wang, Jin Cui, Fumiyo Fukumoto, Yoshimi SuzukiEMNLP 2024 · DOI

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in linguistic reasoning and zero-shot tasks. However, their application in recommendation systems is often hindered by a lack of structural awareness. Traditional recommenders rely on the user-item interaction graph to identify patterns, whereas LLMs typically treat recommendation as a text-generation problem. ELMRec (Enhanced LLM-based Recommender) bridges this gap by injecting interaction graph-aware embeddings into the LLM, significantly improving performance without requiring expensive graph pre-training.

The Challenge: High-order Signals and ID Fragmentation

Two core issues limit standard LLMs in recommendation tasks. First, they struggle to capture high-order collaborative signals. In a recommendation context, these are multi-hop relationships (e.g., User A likes Item X, User B also likes Item X and Item Y; therefore, User A might like Item Y).

Visualizing multi-hop relationships in a user-item interaction graph.

Second, LLM tokenizers often fragment user and item IDs into meaningless subwords (e.g., user_1234 becomes user, _, 12, 34), which dilutes their representational power. Furthermore, the authors identified a "recency bias" where models often hallucinate recommendations based on older user history rather than the most relevant recent interactions.

Methodology: The ELMRec Framework

ELMRec introduces two primary innovations to address these limitations:

1. Interaction Graph-aware Whole-word Embeddings

Instead of relying on subword tokens, ELMRec represents each user and item ID with a whole-word embedding. These embeddings are made "graph-aware" through a process called Random Feature Propagation (RFP).

The ELMRec framework: Integrating graph-aware whole-word embeddings into a Large Language Model via Random Feature Propagation.

In this approach, user and item vectors are initialized with random noise and then propagated through the interaction graph using LightGCN. This ensures that nodes that are "close" in the real-world graph end up with similar embedding vectors. These vectors are then added to the LLM's standard input, providing the model with a spatial sense of the global network structure.

2. Training-free Reranking

To solve the recency gap, ELMRec employs a plug-and-play reranking strategy during inference. The model generates a larger set of candidates ($k+N$) and their associated probabilities. It then filters out previously interacted items and reranks the top candidates, ensuring the focus remains on the most probable new items.

Key Insights and Results

One of the most significant findings is that complex GNN pre-training is not required; simple random feature propagation provides sufficient collaborative signals. Mathematically, the authors prove that adding whole-word embeddings increases the rank of the LLM's attention matrix, preventing ID "collisions" and allowing the model to better distinguish between millions of unique entities.

In empirical evaluations across Amazon benchmark datasets (Sports, Beauty, and Toys), ELMRec achieved substantial gains:

  • Direct Recommendation: Outperformed the runner-up GNN (NCL) by 15.9% to 53.1%.
  • Sequential Recommendation: Consistently beat the state-of-the-art POD model by 2.7% to 13.5%.

Conclusion and Future Work

ELMRec successfully combines the reasoning power of LLMs with the structural intelligence of graphs. While the current implementation faces some scalability challenges due to the $O(n^2)$ complexity of LightGCN and typical cold-start issues, it represents a major step forward. Future research will explore more efficient graph architectures and more sophisticated, learnable reranking methods.

Summary updated

BibTeX
@inproceedings{wang2024enhancing,
  author = {Xinfeng Wang and Jin Cui and Fumiyo Fukumoto and Yoshimi Suzuki},
  title = {{Enhancing High-order Interaction Awareness in LLM-based Recommender Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.653},
}
Figure