TL;DR
ReuseGR is a novel framework that decouples generative retrieval into offline query-level "planning" and online request-level "completion." By caching query-dependent Semantic ID (SID) prefixes generated by a 7B model and completing them with a 0.5B model conditioned on user history, it achieves the retrieval quality of a large model with the serving efficiency of a small one—reducing online latency by nearly 40% while boosting click-through rates in production.
What Problem Does This Paper Address?
Generative retrieval is a rising paradigm where models directly predict item identifiers (Semantic IDs) as token sequences, rather than matching vectors in a latent space. While larger models (e.g., 7B parameters) offer superior retrieval quality, their autoregressive decoding is too slow for the millisecond-latency requirements of industrial search engines.
The authors identify a key redundancy: when different users search for the same query, the initial "planning" of which item categories to explore is often identical. However, their unique interaction histories mean they should ultimately see different items. Traditional systems either recompute everything online (slow) or cache complete results (not personalized). ReuseGR aims to find the "sweet spot" by reusing shared prefixes while keeping the final steps of generation personalized.
Methods
ReuseGR splits the retrieval process into two distinct phases:
- Offline Planning (7B Model): For a given query, a large 7B model (CellPlan) generates a set of SID prefixes. These prefixes represent "subtrees" of potential items.
- Online Completion (0.5B Model): A smaller 0.5B model (LeafFill) takes these cached prefixes and the specific user's history to generate the remaining tokens (suffixes), completing the SIDs.
Key Innovations:
- ReuseCut (Adaptive Prefix Caching): Not all queries are created equal. Some benefit from deep caching, while others need more online flexibility. ReuseCut uses a classifier to predict the optimal "cache depth" for each query based on historical coverage.
- Path-Level Knowledge Distillation (Path KD): To ensure the 0.5B model doesn't lose the "wisdom" of the 7B model, the authors distill the 7B model's suffix-scoring knowledge into the 0.5B model.
- BudgetCut: Since the model generates many potential "branches," BudgetCut uses a policy head (optimized via Group-Relative Policy Optimization, or GRPO) to select the most promising branches to complete within a fixed computational budget.
Datasets
The researchers evaluated ReuseGR on AE-PV, an industrial search-advertising dataset containing:
- 3.38M catalog items.
- Approximately 2M query-item records.
- 44,225 test queries.
They also conducted a 14-day online A/B test on a major global e-commerce platform serving hundreds of millions of users.
Key Insights and Ideas
- The "Early Layer" Advantage: Analysis showed that query information contributes significantly more to the first two layers of SID generation than to later layers. This empirically justifies why sharing prefixes across users is so effective.
- Adaptive Depth is Essential: Using a fixed cache depth for all queries is suboptimal. Adaptive depth selection (ReuseCut) improved Hit Rate (HR@100) by 3.17 points compared to permuted assignments with the same average depth.
- Personalization Recovery: Even though the prefixes are query-only, the history-conditioned completion successfully recovers relevant items that were not in the 7B model's top-100 query-only list for 5.67% of requests.
Key Results
ReuseGR delivers a rare "win-win" in system design: better quality and lower latency.
- Efficiency: Compared to an all-online 0.5B baseline, ReuseGR reduces mean online latency by 38.9% (from 117ms to 71.45ms). Compared to a 7B baseline, it is 57.7% faster.
- Quality: It achieves a Hit Rate (HR@100) of 85.82%, which is comparable to the 7B teacher model (85.71%) and significantly higher than the 0.5B baseline (81.99%).
- Business Impact: In the 14-day A/B test, ReuseGR achieved a 0.93% gain in Click-Through Rate (CTR) and a 1.05% gain in Revenue Per Thousand requests (RPM) over the 0.5B baseline.
Limitations and Future Work
- Cold Start for Queries: While ReuseCut can predict depths for new queries using text features, the system still relies on an asynchronous 7B planning queue for "cold" queries. Immediate requests for brand-new queries fall back to the 0.5B E2E model.
- Cache Maintenance: The paper notes that a 10% daily refresh of the 1M-entry cache costs about 2.44 A100 GPU-hours. While manageable, this adds an operational overhead compared to purely online systems.
- Future Directions: The authors suggest that the decoupled architecture could be further optimized by exploring different SID structures or more advanced distillation techniques for the online scorer.
Summary updated