Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation

Rong Shan, Jiachen Zhu, Jianghao Lin, Chenxu Zhu, Bo Chen, Ruiming Tang, Yong Yu, Weinan Zhang

Introduction

In the evolving landscape of recommendation systems (RecSys), the integration of Large Language Models (LLMs) has promised a leap in semantic understanding. However, researchers have identified a significant bottleneck: "lifelong sequential behavior incomprehension." Unlike traditional recommendation models such as SIM, which improve as user history grows, LLM-based recommenders often see a performance decline once a user's history exceeds approximately 15 items. This phenomenon occurs even when the total token count is well within the model's context window, suggesting a reasoning failure rather than a memory limitation.

To address this, the authors propose ReLLaX (Retrieval-enhanced Large Language models Plus), a full-stack framework that optimizes the recommendation pipeline across three critical dimensions: data, prompts, and parameters. By bridging the gap between textual semantics and collaborative filtering, ReLLaX enables LLMs to effectively process long-term user behaviors.

The Problem: Lifelong Sequential Behavior Incomprehension

Traditional RecSys models are built to handle long sequences of item IDs, extracting collaborative signals that grow stronger with more data. In contrast, LLMs process these histories as textual descriptions. As these sequences grow longer and more heterogeneous, they become increasingly "noisy." The LLM struggles to distinguish between a user's core preferences and transient "random walk" interactions. This results in a performance peak at short sequence lengths, followed by a steady decline—a stark contrast to the "more is better" scaling seen in traditional systems.

The ReLLaX Framework

ReLLaX introduces a three-tiered optimization strategy to transform LLMs into robust recommenders:

1. Data Level: Semantic User Behavior Retrieval (SUBR)

Rather than feeding the LLM a simple chronological list of recent items, ReLLaX employs Semantic User Behavior Retrieval (SUBR). This method uses the LLM itself to encode items into semantic vectors, retrieving only the historical items most relevant to the current target item. By filtering out irrelevant noise, SUBR ensures the model's context is populated with high-signal data, regardless of when the interaction occurred.

2. Prompt Level: Soft Prompt Augmentation (SPA)

LLMs excel at understanding text but lack the "collaborative knowledge" inherent in traditional ID-based models (e.g., knowing that users who liked Item A also frequently liked Item B). Soft Prompt Augmentation (SPA) addresses this by taking item embeddings from a pretrained traditional model and projecting them into "soft tokens." These tokens are appended to the text prompt, providing the LLM with a collaborative "cheat sheet" that complements its semantic world knowledge.

3. Parameter Level: Component Fully-interactive LoRA (CFLoRA)

Standard Low-Rank Adaptation (LoRA) can be too rigid for the complex, multi-modal nature of sequential recommendation. ReLLaX introduces Component Fully-interactive LoRA (CFLoRA), which decomposes LoRA matrices into "atom components." These components are allowed to interact fully through a learned interaction matrix guided by the user's representation. This makes the model significantly more expressive and better equipped to capture long-range dependencies in user behavior.

Experimental Results

The framework was evaluated on three major benchmarks: BookCrossing, MovieLens-1M, and MovieLens-25M. The results demonstrated several key breakthroughs:

  • State-of-the-Art Performance: ReLLaX consistently outperformed both traditional ID-based models (like DeepFM and DIN) and existing LLM-based methods (like TALLRec and CoLLM).
  • Solving the Incomprehension Gap: Unlike vanilla LLMs, ReLLaX showed steady performance improvements as sequence lengths increased, successfully mimicking the behavior of specialized lifelong sequential models.
  • Data Efficiency: ReLLaX achieved these results using less than 10% of the training data required by traditional models, leveraging the pre-existing knowledge within the LLM.
  • Attention Accuracy: Case studies revealed that ReLLaX’s attention mechanisms focused more accurately on relevant historical items (e.g., matching genres) rather than being distracted by the most recent, but potentially irrelevant, clicks.

Conclusion and Future Directions

ReLLaX demonstrates that the "incomprehension" of long sequences in LLMs is not an inherent limitation of the architecture, but a challenge of data representation and model adaptation. By unifying semantic and collaborative signals, ReLLaX sets a new standard for LLM-based recommendation.

The primary remaining challenge is inference latency. Due to the computational demands of the underlying Vicuna-13B model, ReLLaX is currently best suited for the re-ranking stage of a recommendation pipeline. Future research will focus on optimizing inference through model distillation and pruning, potentially allowing these full-stack optimizations to be applied to high-throughput candidate generation.

Summary updated

Abstract

As large language models (LLMs) achieve remarkable success in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and are being actively explored currently. In this article, we focus on adapting and enhancing large language models for recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation realms, i.e., LLMs fail to effectively extract useful information from a pure textual context of long user behavior sequence, even if the length of context is well below the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely, R etrieval- e nhanced L arge La nguage models Plus (ReLLaX), which provides full-stack optimization from three perspectives, i.e., data, prompt, and parameter. For data-level enhancement, we design semantic user behavior retrieval (SUBR) to reduce the heterogeneity of the behavior sequence, thus lowering the difficulty for LLMs to extract the essential information from user behavior sequences. Although SUBR can improve the data quality, further increase in the sequence length will still raise its heterogeneity to a level where LLMs can no longer comprehend it. Hence, we further propose to perform prompt-level and parameter-level enhancement, with the integration of conventional recommendation models (CRMs). As for prompt-level enhancement, we apply soft prompt augmentation (SPA) to explicitly inject collaborative knowledge from CRMs into the prompt. The item representations of LLMs are thus more aligned with recommendation, helping LLMs better explore the item relationships in the sequence and facilitating comprehension. Finally, for parameter-level enhancement, we propose component fully-interactive LoRA (CFLoRA). By enabling sufficient interaction between the LoRA atom components, the expressive ability of LoRA is extended, making the parameters effectively capture more sequence information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e., from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments are conducted on three real-world public datasets to demonstrate the superiority of ReLLaX compared with existing baseline models, as well as its capability to alleviate lifelong sequential behavior incomprehension. Our code is available. 1
BibTeX
@article{shan2025full,
  author = {Rong Shan and Jiachen Zhu and Jianghao Lin and Chenxu Zhu and Bo Chen and Ruiming Tang and Yong Yu and Weinan Zhang},
  title = {{Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation}},
  journal = {ACM Transactions on Recommender Systems},
  year = {2025},
  doi = {10.1145/3766553},
}
Figure