KuaiRecV2: Benchmarking Large-Scale Continual Learning for Diversified and Multi-task Recommendation

Chenxu Li, Shuchang Liu, Hantao Shu, Wei Yuan, Zhe Xu, Guoqing Hu, Yan Wang, Biao Yang, Xiang Li, Cheng Ling, Fan Yang, Yongqi Liu, Lantao Hu, Han Li, Kaiqiao Zhan, Kun Gai, An Zhang, Xiang WangarXiv 2026 · arXiv

TL;DR

KuaiRecV2 is a massive new benchmark designed to bridge the gap between offline recommendation research and real-world industrial performance. By providing a 30-day, million-scale dataset from Kuaishou, it challenges models to handle continual learning under dynamic item pools, combinatorial diversity at the token level, and multi-task reward balancing.

What Problem Does This Paper Address?

In the world of recommender systems, researchers often face a frustrating "offline-online gap": a model that looks like a superstar on a static dataset often fails to deliver results when deployed in a live environment. The authors identify three primary culprits for this inconsistency:

  1. The Continual Learning Challenge: Real-world systems aren't static. New items (like short videos) are uploaded constantly, and user interests shift. Most benchmarks ignore this "streaming" reality, leading to models that can't adapt or that suffer from catastrophic forgetting.
  2. The Combinatorial Diversity Challenge: Modern generative recommenders (which treat items like sequences of tokens) struggle with the accuracy-diversity trade-off. Diversity isn't just about the final item; it's about the paths taken in the "token space."
  3. The Multi-Task Reward Balancing Challenge: Systems must balance multiple goals (e.g., clicks, likes, watch time). Generative models are great at predicting sequences but are notoriously hard to control for these specific, often conflicting, business demands.

Methods

The paper advocates for LLM-style generative recommendation, where items are represented as sequences of semantic tokens (Semantic IDs). The authors implement a long-term recommendation pipeline characterized by:

  • Consecutive Daily Training: Models are updated daily using new interaction data, mimicking a production environment.
  • Dynamic Candidate Pools: A sliding-window strategy removes old items and adds new ones, forcing the model to handle "cold-start" scenarios continually.
  • RL-based Multi-Task Alignment: Beyond standard Supervised Fine-Tuning (SFT), they use Reinforcement Learning (RL) to align model outputs with complex user feedback signals (likes, follows, etc.).

Datasets

The authors introduce KuaiRecV2, a benchmark built on industrial data from Kuaishou:

  • KuaiRecV2-FullMonth: Contains 484 million interaction records from 124k users over 30 days, involving nearly 25 million items.
  • KuaiRecV2-Small: A subset of the first two days for rapid model validation.
  • Rich Metadata: Includes 64-dimensional multi-modal embeddings, 8-level semantic IDs, and MLLM-inferred tags for every item.
Figure 1: Data Organization of KuaiRecV2.

Key Insights and Ideas

  • Catastrophic Forgetting is Real: Even with advanced generative models, performance on "Day 1" data drops sharply after training on subsequent days. Standard techniques like Replay buffers help but don't solve the fundamental problem of retaining long-term user preferences while adapting to new trends.
  • The RL Stability Trap: While RL can improve multi-task performance in the short term, the authors found that RL-only strategies can be fragile. Without the "anchor" of SFT, models can sharply degrade when the item pool shifts significantly (see Figure 4/6).
  • Diversity Bottlenecks: Analysis of token-level diversity reveals that most "variety" in recommendations comes from the first few tokens of an item's ID. Later tokens become highly biased, suggesting that current generative recommenders have a "diversity ceiling" that requires more sophisticated decoding strategies.
  • Architecture > Size: For generative recommenders, the choice of architecture (e.g., using semantic IDs vs. traditional IDs) is often more critical for scaling than simply increasing the number of parameters.

Key Results

The benchmark evaluates several state-of-the-art models, including MiniOneRec (0.5B parameters) and TIGER.

  • Scalability: Generative sequential models significantly outperform traditional models like SASRec and GRURec on large-scale item pools. MiniOneRec achieved the highest NDCG and HitRate across all tested timeframes.
  • Adaptation Spikes: Figure 3/5 illustrates the "learning pain" of continual updates. Every time the item pool changes at a day boundary, there is a sharp spike in training loss as the model struggles to integrate new content.
Figure 3: Training loss during 5-day continual learning. Orange squares indicate the initial loss at each day bound- ary (when item pools change), and green dots show the final loss after training. The inset shows the rapid initial convergence on Day 1.
  • Multi-task Gains: Alternating between SFT and RL proved to be the most robust strategy for balancing multiple user behaviors, outperforming RL-only approaches which failed to maintain stability over time.
Figure 4: HR@50 across training days. We compare the results of alternating SFT and RL with those of RL-only.

Limitations and Future Work

The authors acknowledge that KuaiRecV2, while realistic, still lacks some industrial complexities:

  • Privacy Constraints: User profile features are omitted for privacy, which limits the study of user-side cold-starts.
  • Cross-Domain Gaps: The benchmark focuses on a single platform (Kuaishou) and does not address how models might transfer knowledge between different recommendation domains.
  • Future Directions: The team suggests focusing on token-to-item decoders that better align semantic generation with final ranking, and investigating scaling laws specifically for the continual learning setting.

Summary updated

Abstract

Recommender systems, which retrieve items of interest to users based on their interaction histories, have recently witnessed several key technological advancements, benefiting from the semantic representation space and model-data scaling. Nevertheless, practitioners and researchers have frequently encountered critical inconsistencies between simple offline evaluations and complex online results, and this gap still exists in the era of large models. In this work, we address several key factors that contribute to this inconsistency: 1) the \textit{large-scale continual learning challenge} that assumes a practical demand to maintain long-term model effectiveness under dynamic item pools and distributional shifts; 2) the \textit{combinatorial diversity challenge} that requires the generation strategy to solve the accuracy-diversity trade-off in both the finer token level and the holistic item level; and 3) the \textit{multi-task reward balancing challenge}, which is particularly critical for modern generative models that can accurately model the sequence distribution but are less controllable for the multi-task demands. To address these challenges and further pave the way for the development of more realistic recommendation solutions, we introduce KuaiRecV2, a deliberately constructed benchmark with industrial-level datasets from Kuaishou. Specifically, we provide a large-scale interaction dataset with million-scale multi-modal item information and long-term interaction records across 30 days, with a dynamic video candidate pool. Additionally, we provide a rigorous benchmark that evaluates retrieval accuracy, diversity, and multi-task performance in one rubric system, where an effective advancement occurs if and only if the new method improves all metrics. The code is available at https://github.com/cxli2333/KuaiRecV2
BibTeX
@misc{li2026kuairecv,
  author = {Chenxu Li and Shuchang Liu and Hantao Shu and Wei Yuan and Zhe Xu and Guoqing Hu and Yan Wang and Biao Yang and Xiang Li and Cheng Ling and Fan Yang and Yongqi Liu and Lantao Hu and Han Li and Kaiqiao Zhan and Kun Gai and An Zhang and Xiang Wang},
  title = {{KuaiRecV2: Benchmarking Large-Scale Continual Learning for Diversified and Multi-task Recommendation}},
  year = {2026},
  eprint = {2611.16795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16795},
}
Figure