CRIM: Composable and Reusable Item Memory for Efficient Personalized Retrieval

Haolin Li, Haipeng Zhang, Mang Li, Zhaohui Xia, Xun Zhu, Miao Li, Yu Zhang, Ji WuarXiv 2026 · arXiv

TL;DR

CRIM (Composable and Reusable Item Memory) is a new framework that drastically reduces the cost of using Large Language Models (LLMs) for personalized search and recommendation. By compressing item text into reusable "summary tokens" and decoupling item semantics from user history modeling, CRIM achieves a 4.05x latency speedup and an 80% reduction in serving input length while outperforming state-of-the-art dense retrieval baselines.

What Problem Does This Paper Address?

Modern personalized retrieval systems—like those powering Amazon or Netflix—rely on a user's interaction history to predict what they want next. While LLMs are excellent at understanding the rich semantics of item descriptions, they are computationally expensive.

The authors identify two major inefficiencies in current LLM-based approaches:

  1. Redundant Encoding: The same item (e.g., a popular movie) is encoded from scratch thousands of times across different users and requests.
  2. Contextual Bloat: As user histories grow, the input sequences to the LLM become massive, leading to quadratic increases in attention costs and high inference latency.

Existing solutions often choose between two extremes: caching a single static vector for an item (which loses contextual nuance) or re-encoding everything every time (which is too slow). CRIM aims to find the "sweet spot" by creating compressed, reusable memories that can still be composed dynamically to reflect a user's evolving interests.

Methods

CRIM introduces a two-module architecture:

  • Item Memory Encoder (IME): This module takes an item's text and ID and compresses them into a small, fixed number of continuous vectors called summary tokens ($K=4$ in their experiments). These tokens are independent of specific users and can be precomputed and cached.
  • Request Encoder (RE): At inference time, the RE retrieves these cached summary tokens for every item in a user's history, composes them in chronological order, and integrates an optional search query to produce a final personalized representation.

To ensure these compressed memories are actually useful for sequence modeling, the authors use REC tokens (readout tokens). These are inserted at intervals throughout the history to provide intermediate supervision—essentially forcing the model to predict the next item based on the compressed history prefix.

Figure 2: Overview of CRIM. The IME encodes each item into reusable continuous memories. The RE composes cached memory blocks in interaction order, incorporates an optional query, and produces a representation for retrieval against separately encoded catalog candidates.

A key innovation is the Joint Training mechanism. To keep the IME and RE aligned without the massive cost of re-encoding full histories during training, CRIM uses a "sampled live re-encoding" strategy. For each training request, only a few items are re-encoded "live" to provide gradients to the IME, while the rest are pulled from a stale cache. Periodic refreshes keep the cache synchronized with the evolving model.

Datasets

The authors evaluated CRIM on three datasets:

  • MovieLens-1M (ML1M): A classic benchmark for sequential recommendation.
  • Qilin: A recent multimodal dataset for query-conditioned retrieval.
  • LazSeq (New): A new benchmark introduced in this paper specifically for long-history, query-conditioned retrieval. It features histories of up to 500 interactions and a catalog of nearly 50,000 items, providing a more realistic testbed for industrial-scale retrieval.

Key Insights and Ideas

  • Separation of Concerns: By separating stable item semantics (what is this item?) from dynamic request context (how does this item fit this user's current need?), CRIM allows for massive reuse of computation.
  • Compression is not just for storage: Using summary tokens isn't just about saving space; it's about reducing the "serving length" of the input. This directly translates to faster inference because the LLM's attention mechanism has fewer positions to process.
  • Predictive Supervision: The use of REC tokens ensures that the compressed memories don't just capture what an item is, but also how it functions as a signal in a sequence of behaviors.
  • Amortized Efficiency: The authors provide a detailed cost analysis showing a "break-even" point. While CRIM has higher upfront offline costs for cache construction, it becomes significantly cheaper than standard dense encoders after just ~31,500 requests.

Key Results

CRIM demonstrates that you don't have to sacrifice accuracy for speed:

  • Performance: On the new LazSeq dataset, CRIM achieved the highest Recall@100 (0.6969), outperforming the strong Qwen3-Embedding-4B baseline.
  • Latency: It achieved a 4.05x speedup in end-to-end latency compared to the dense reference model (90.76 ms vs 367.45 ms).
  • Token Efficiency: The mean serving input length was reduced by 80.75%, dropping from over 5,000 tokens to just 967.
Figure 1: Recall@100 versus mean request latency on LazSeq (single A100, histories up to 500 interactions). The inset re- ports mean serving input lengths, including memory slots, REC tokens, queries, and special tokens.

Limitations and Future Work

  • Cache Maintenance: While efficient, the system requires a strategy for periodic cache refreshes to prevent "representation staleness" as the model continues to learn.
  • Storage Overhead: Storing $K$ vectors per item (plus candidate vectors) requires more storage than single-vector embedding models, though the authors argue this is a worthwhile trade-off for the gains in retrieval quality and inference speed.
  • Future Directions: The authors suggest exploring more sophisticated readout mechanisms and testing the framework on even larger-scale industrial catalogs with millions of items.

Summary updated

Abstract

Large language models incur substantial training and inference costs in personalized search and recommendation, as growing interaction histories lengthen request inputs and repeatedly expose the same item text to the model. We propose \textbf{C}omposable and \textbf{R}eusable \textbf{I}tem \textbf{M}emory (CRIM), a framework that separates item-level semantic encoding from request-level history modeling. A small set of learnable summary tokens compresses each item into compact continuous memories, while segment-level readout tokens (REC tokens) introduce predictive supervision within the interaction history. These memories are independent of users and queries and can therefore be reused across requests. At inference time, CRIM composes cached memories in interaction order and incorporates an optional query to produce a personalized retrieval representation, reducing repeated item-text encoding while retaining request-dependent contextual modeling. We further introduce a joint training mechanism that combines cached memories with sampled live re-encoding. Re-encoding only a few history items per request allows the retrieval objective to jointly optimize the item and request encoders at limited computational cost, while periodic cache refreshes maintain consistency. Extensive evaluation and cost analysis on MovieLens-1M, Qilin, and our new LazSeq dataset show that CRIM achieves state-of-the-art retrieval performance with substantially lower inference costs at the same backbone scale, while limiting item re-encoding during joint training. We will publicly release our code and dataset.
BibTeX
@misc{li2026crim,
  author = {Haolin Li and Haipeng Zhang and Mang Li and Zhaohui Xia and Xun Zhu and Miao Li and Yu Zhang and Ji Wu},
  title = {{CRIM: Composable and Reusable Item Memory for Efficient Personalized Retrieval}},
  year = {2026},
  eprint = {2610.35323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35323},
}
Figure