Abstract
In large-scale recommender systems, ultra-long user behavior sequences encode rich signals of evolving interests. Extending sequence length generally improves accuracy, but directly modeling such sequences in production is infeasible due to latency and memory constraints. Existing solutions fall into two categories: (1) top-k retrieval, which truncates the sequence and may discard most attention mass when L >> k; and (2) encoder-based compression, which preserves coverage but often over-compresses and fails to incorporate key context such as temporal gaps or target-aware signals. Neither class achieves a good balance of low-loss compression, context awareness, and efficiency. We propose VQL, a context-aware Vector Quantization Attention framework for ultra-long behavior modeling, with three innovations. (1) Key-only quantization: only attention keys are quantized, while values remain intact; we prove that softmax normalization yields an error bound independent of sequence length, and a codebook loss directly supervises quantization quality. This also enables L-free inference via offline caches. (2) Multi-scale quantization: attention heads are partitioned into groups, each with its own small codebook, which reduces quantization error while keeping cache size fixed. (3) Efficient context injection: static features (e.g., item category, modality) are directly integrated, and relative position is modeled via a separable temporal kernel. All context is injected without enlarging the codebook, so cached representations remain query-independent. Experiments on three large-scale datasets (KuaiRand-1K, KuaiRec, TMALL) show that VQL consistently outperforms strong baselines, achieving higher accuracy while reducing inference latency, establishing a new state of the art in balancing accuracy and efficiency for ultra-long sequence recommendation.
Introduction
In modern recommendation engines, a user’s interaction history is a goldmine for understanding evolving interests. However, modeling "ultra-long" sequences (e.g., $L > 5000$) creates a massive computational bottleneck. Existing methods often struggle to balance three competing goals: low-cost inference, context-aware modeling, and low-loss compression.
Current solutions typically fall into two flawed categories:
- Top-k Retrieval (e.g., SIM, TWIN): These select a subset of relevant items, potentially discarding significant attention mass and missing subtle long-term patterns.
- Encoder-based Compression (e.g., MIMN): These squash sequences into fixed-size vectors, often losing fine-grained details and failing to incorporate critical context like temporal gaps.
Methodology: Vector Quantization Attention (VQL)
VQL addresses these limitations by reimagining the attention mechanism through Vector Quantization (VQ). The framework is designed to achieve constant-time inference relative to sequence length while maintaining high fidelity.
Key Innovations
- Key-only Quantization: Unlike standard VQ-VAE approaches that quantize entire embeddings, VQL quantizes only the attention Keys ($K$) into a learnable codebook. Values ($V$) remain intact. This allows for a "one-hot extraction identity" where Values can be pre-aggregated into a cache offline. Consequently, online inference cost becomes independent of the original sequence length $L$. Mathematically, the authors prove that the quantization error bound remains independent of sequence length, ensuring stability for ultra-long modeling.
- Multi-scale Quantization: To minimize quantization error, VQL employs Grouped Vector Quantization (GVQ). Attention heads are partitioned into groups, each with its own small codebook. This multi-scale approach captures more nuanced user behavior without significantly increasing cache size.
- Efficient Context Injection: VQL integrates static features and relative temporal gaps without enlarging the codebook. By using a separable temporal kernel, time-decay factors are incorporated into the offline cache, ensuring the online step remains "Query-free" and efficient.
Experiments and Results
VQL was evaluated on three large-scale datasets: KuaiRand-1K, KuaiRec, and TMALL. These datasets feature average sequence lengths ranging from hundreds to thousands of interactions, with the 90th percentile reaching nearly 9,000 interactions in KuaiRand.
Performance and Efficiency
- Accuracy: VQL consistently outperformed state-of-the-art baselines. On KuaiRand-1K, VQL achieved an AUC of 71.47%, surpassing the next best model, LREA (70.64%).
- Latency: In production-simulated environments, VQL's inference latency remained nearly constant as sequence length increased, whereas retrieval-based models like SIM showed linear growth in latency.
- Robustness: VQL demonstrated stable performance gains as sequence length scaled, proving its ability to effectively utilize ultra-long histories where other models plateau or degrade due to information loss.
Conclusion
VQL introduces a novel approach to ultra-long user behavior modeling by combining the efficiency of vector quantization with the flexibility of attention mechanisms. By quantizing only keys and enabling query-independent caching, VQL breaks the linear dependency between sequence length and online latency. This framework establishes a new state-of-the-art for balancing accuracy and efficiency in modern recommendation engines.
Summary updated