ReuseGR: Offline--Online Decoupled Generative Retrieval via Adaptive Prefix Reuse

Xujing Wang, Yufeng Chen, Boxuan Zhang, Chao Wei, Cai Xu, Jie Zhao, Weiru Zhang, Wei Zhao, Ziyu GuanarXiv 2026 · arXiv

TL;DR

ReuseGR is a novel framework that decouples generative retrieval into offline query-level "planning" and online request-level "completion." By caching query-dependent Semantic ID (SID) prefixes generated by a 7B model and completing them with a 0.5B model conditioned on user history, it achieves the retrieval quality of a large model with the serving efficiency of a small one—reducing online latency by nearly 40% while boosting click-through rates in production.

What Problem Does This Paper Address?

Generative retrieval is a rising paradigm where models directly predict item identifiers (Semantic IDs) as token sequences, rather than matching vectors in a latent space. While larger models (e.g., 7B parameters) offer superior retrieval quality, their autoregressive decoding is too slow for the millisecond-latency requirements of industrial search engines.

The authors identify a key redundancy: when different users search for the same query, the initial "planning" of which item categories to explore is often identical. However, their unique interaction histories mean they should ultimately see different items. Traditional systems either recompute everything online (slow) or cache complete results (not personalized). ReuseGR aims to find the "sweet spot" by reusing shared prefixes while keeping the final steps of generation personalized.

Methods

ReuseGR splits the retrieval process into two distinct phases:

  1. Offline Planning (7B Model): For a given query, a large 7B model (CellPlan) generates a set of SID prefixes. These prefixes represent "subtrees" of potential items.
  2. Online Completion (0.5B Model): A smaller 0.5B model (LeafFill) takes these cached prefixes and the specific user's history to generate the remaining tokens (suffixes), completing the SIDs.
The ReuseGR architecture: Offline planning with a 7B model and online completion with a 0.5B model.

Key Innovations:

  • ReuseCut (Adaptive Prefix Caching): Not all queries are created equal. Some benefit from deep caching, while others need more online flexibility. ReuseCut uses a classifier to predict the optimal "cache depth" for each query based on historical coverage.
  • Path-Level Knowledge Distillation (Path KD): To ensure the 0.5B model doesn't lose the "wisdom" of the 7B model, the authors distill the 7B model's suffix-scoring knowledge into the 0.5B model.
  • BudgetCut: Since the model generates many potential "branches," BudgetCut uses a policy head (optimized via Group-Relative Policy Optimization, or GRPO) to select the most promising branches to complete within a fixed computational budget.

Datasets

The researchers evaluated ReuseGR on AE-PV, an industrial search-advertising dataset containing:

  • 3.38M catalog items.
  • Approximately 2M query-item records.
  • 44,225 test queries.

They also conducted a 14-day online A/B test on a major global e-commerce platform serving hundreds of millions of users.

Key Insights and Ideas

  • The "Early Layer" Advantage: Analysis showed that query information contributes significantly more to the first two layers of SID generation than to later layers. This empirically justifies why sharing prefixes across users is so effective.
Layer-wise analysis showing that query-specific gains are highest in the early layers of SID generation.
  • Adaptive Depth is Essential: Using a fixed cache depth for all queries is suboptimal. Adaptive depth selection (ReuseCut) improved Hit Rate (HR@100) by 3.17 points compared to permuted assignments with the same average depth.
  • Personalization Recovery: Even though the prefixes are query-only, the history-conditioned completion successfully recovers relevant items that were not in the 7B model's top-100 query-only list for 5.67% of requests.

Key Results

ReuseGR delivers a rare "win-win" in system design: better quality and lower latency.

  • Efficiency: Compared to an all-online 0.5B baseline, ReuseGR reduces mean online latency by 38.9% (from 117ms to 71.45ms). Compared to a 7B baseline, it is 57.7% faster.
  • Quality: It achieves a Hit Rate (HR@100) of 85.82%, which is comparable to the 7B teacher model (85.71%) and significantly higher than the 0.5B baseline (81.99%).
  • Business Impact: In the 14-day A/B test, ReuseGR achieved a 0.93% gain in Click-Through Rate (CTR) and a 1.05% gain in Revenue Per Thousand requests (RPM) over the 0.5B baseline.
Figure 1: Query-prefix reuse in ReuseGR. (a) Queries reuse different prefix lengths (one and three tokens shown, dashed) and use their histories to complete suffixes. Cache depths range from zero to three. (b) Complete-SID HR@100 and mean online latency on AE-PV (44,225 queries, one A100, batch size 1), with cached prefixes available for ReuseGR.

Limitations and Future Work

  • Cold Start for Queries: While ReuseCut can predict depths for new queries using text features, the system still relies on an asynchronous 7B planning queue for "cold" queries. Immediate requests for brand-new queries fall back to the 0.5B E2E model.
  • Cache Maintenance: The paper notes that a 10% daily refresh of the 1M-entry cache costs about 2.44 A100 GPU-hours. While manageable, this adds an operational overhead compared to purely online systems.
  • Future Directions: The authors suggest that the decoupled architecture could be further optimized by exploring different SID structures or more advanced distillation techniques for the online scorer.

Summary updated

Abstract

Larger generative retrieval models can improve retrieval quality, but their decoding cost makes low-latency serving difficult. When different users submit the same query, generating complete Semantic IDs (SIDs) for every request repeats computation, while their interaction histories call for different results. The challenge is to reuse large-model computation across requests while keeping retrieval personalized. We propose \textbf{ReuseGR}, which separates query-level candidate planning from request-level personalized completion. A 7B offline planner generates SID prefix sets shared across requests with the same query. A 0.5B online model uses each request's history to select prefixes and generate the remaining tokens. ReuseCut chooses a cache depth for each query based on historical coverage, balancing computation reuse and candidate coverage. To strengthen online completion, we transfer 7B suffix-scoring knowledge to 0.5B through path-level knowledge distillation (Path KD). BudgetCut then selects scored branches to improve final top-100 recall under a fixed branch budget. Its selection policy is initialized by on-policy distillation (OPD) and refined through group-relative policy optimization (GRPO). On an industrial search-advertising dataset with approximately 2M query--item records and 3.38M catalog items, ReuseGR improves query-level HR@100 by 3.83 percentage points and reduces mean online latency by 38.9\% relative to an all-online 0.5B baseline when prefixes are prepared. It achieves HR@100 comparable to an all-online 7B baseline with 57.7\% lower online latency. A 14-day user-level online A/B test against the 0.5B baseline shows relative gains of 0.93\% in click-through rate and 1.05\% in revenue per thousand requests.
BibTeX
@misc{wang2026reusegr,
  author = {Xujing Wang and Yufeng Chen and Boxuan Zhang and Chao Wei and Cai Xu and Jie Zhao and Weiru Zhang and Wei Zhao and Ziyu Guan},
  title = {{ReuseGR: Offline--Online Decoupled Generative Retrieval via Adaptive Prefix Reuse}},
  year = {2026},
  eprint = {2611.04259},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04259},
}
Figure