TL;DR
KuaiRecV2 is a massive new benchmark designed to bridge the gap between offline recommendation research and real-world industrial performance. By providing a 30-day, million-scale dataset from Kuaishou, it challenges models to handle continual learning under dynamic item pools, combinatorial diversity at the token level, and multi-task reward balancing.
What Problem Does This Paper Address?
In the world of recommender systems, researchers often face a frustrating "offline-online gap": a model that looks like a superstar on a static dataset often fails to deliver results when deployed in a live environment. The authors identify three primary culprits for this inconsistency:
- The Continual Learning Challenge: Real-world systems aren't static. New items (like short videos) are uploaded constantly, and user interests shift. Most benchmarks ignore this "streaming" reality, leading to models that can't adapt or that suffer from catastrophic forgetting.
- The Combinatorial Diversity Challenge: Modern generative recommenders (which treat items like sequences of tokens) struggle with the accuracy-diversity trade-off. Diversity isn't just about the final item; it's about the paths taken in the "token space."
- The Multi-Task Reward Balancing Challenge: Systems must balance multiple goals (e.g., clicks, likes, watch time). Generative models are great at predicting sequences but are notoriously hard to control for these specific, often conflicting, business demands.
Methods
The paper advocates for LLM-style generative recommendation, where items are represented as sequences of semantic tokens (Semantic IDs). The authors implement a long-term recommendation pipeline characterized by:
- Consecutive Daily Training: Models are updated daily using new interaction data, mimicking a production environment.
- Dynamic Candidate Pools: A sliding-window strategy removes old items and adds new ones, forcing the model to handle "cold-start" scenarios continually.
- RL-based Multi-Task Alignment: Beyond standard Supervised Fine-Tuning (SFT), they use Reinforcement Learning (RL) to align model outputs with complex user feedback signals (likes, follows, etc.).
Datasets
The authors introduce KuaiRecV2, a benchmark built on industrial data from Kuaishou:
- KuaiRecV2-FullMonth: Contains 484 million interaction records from 124k users over 30 days, involving nearly 25 million items.
- KuaiRecV2-Small: A subset of the first two days for rapid model validation.
- Rich Metadata: Includes 64-dimensional multi-modal embeddings, 8-level semantic IDs, and MLLM-inferred tags for every item.
Key Insights and Ideas
- Catastrophic Forgetting is Real: Even with advanced generative models, performance on "Day 1" data drops sharply after training on subsequent days. Standard techniques like Replay buffers help but don't solve the fundamental problem of retaining long-term user preferences while adapting to new trends.
- The RL Stability Trap: While RL can improve multi-task performance in the short term, the authors found that RL-only strategies can be fragile. Without the "anchor" of SFT, models can sharply degrade when the item pool shifts significantly (see Figure 4/6).
- Diversity Bottlenecks: Analysis of token-level diversity reveals that most "variety" in recommendations comes from the first few tokens of an item's ID. Later tokens become highly biased, suggesting that current generative recommenders have a "diversity ceiling" that requires more sophisticated decoding strategies.
- Architecture > Size: For generative recommenders, the choice of architecture (e.g., using semantic IDs vs. traditional IDs) is often more critical for scaling than simply increasing the number of parameters.
Key Results
The benchmark evaluates several state-of-the-art models, including MiniOneRec (0.5B parameters) and TIGER.
- Scalability: Generative sequential models significantly outperform traditional models like SASRec and GRURec on large-scale item pools. MiniOneRec achieved the highest NDCG and HitRate across all tested timeframes.
- Adaptation Spikes: Figure 3/5 illustrates the "learning pain" of continual updates. Every time the item pool changes at a day boundary, there is a sharp spike in training loss as the model struggles to integrate new content.
- Multi-task Gains: Alternating between SFT and RL proved to be the most robust strategy for balancing multiple user behaviors, outperforming RL-only approaches which failed to maintain stability over time.
Limitations and Future Work
The authors acknowledge that KuaiRecV2, while realistic, still lacks some industrial complexities:
- Privacy Constraints: User profile features are omitted for privacy, which limits the study of user-side cold-starts.
- Cross-Domain Gaps: The benchmark focuses on a single platform (Kuaishou) and does not address how models might transfer knowledge between different recommendation domains.
- Future Directions: The team suggests focusing on token-to-item decoders that better align semantic generation with final ranking, and investigating scaling laws specifically for the continual learning setting.
Summary updated