TL;DR
TokenFormer V2 is a new unified recommendation architecture that solves the "resolution-complexity dilemma" by decoupling behavior detail from computational cost. By using a group-wise token layout and hierarchical merging, it preserves every individual behavior token throughout the model while achieving state-of-the-art accuracy with 5.1× fewer FLOPs than leading baselines and a 2.33% increase in GMV in live production.
What Problem Does This Paper Address?
Industrial recommendation systems face a frustrating trade-off. To predict what a user wants, you need their history. Long histories provide rich evidence, but modeling them with standard Transformers is computationally ruinous because self-attention cost grows quadratically with sequence length.
To stay within "serving budgets" (the time and compute allowed to generate a recommendation), engineers usually do one of two things:
- Truncate history: Throw away older data.
- Compress history: Use pooling or "summary tokens" to represent groups of actions.
Both approaches lose the "fine-grained evidence" found in specific, individual actions. TokenFormer V2 aims to provide full-resolution modeling—keeping every single behavior event accessible to the final prediction—without the massive computational overhead.
Methods
The core innovation is a local-global decomposition paradigm. Instead of letting every token talk to every other token (dense attention), the model organizes information flow through four key mechanisms:
- Group-wise Token Layout: The behavior sequence is partitioned into contiguous segments. Each segment is bundled with non-sequential context (user/item features) and special "group relay" tokens.
- Local Group Attention: In each layer, tokens only perform dense attention within their small group. This keeps the initial computation linear relative to the total history length.
- Hierarchical Group Merging: As the data moves to deeper layers, adjacent groups merge. This gradually expands the "interaction range" (how far back in time a behavior can "see") without ever pooling or discarding the individual behavior tokens.
- NS Fusion & Global Sync: These modules act as the "highways" for information. NS Fusion mixes non-sequential field features across the whole model, while Global Sync uses a small set of "readout tokens" to coordinate global context across all groups.
- Full-Resolution Readout: Unlike models that predict based on a single summary vector, the final stage of TokenFormer V2 uses cross-attention to look back at every encoded behavior token in the sequence.
Datasets
The authors validated the model across three scales:
- TAAC 2026 Benchmark: The Tencent Advertising Algorithm Competition 2026 dataset, featuring 47.1 million impressions and histories from four distinct behavior domains.
- WeChat Channels (Offline): A massive industrial dataset used to train billion-parameter models on historical advertising logs.
- Live Online A/B Testing: Real-world traffic on WeChat Channels advertising to measure actual business impact (GMV).
Key Insights and Ideas
- Resolution $\neq$ Density: The most important takeaway is that you don't need dense interaction to maintain high resolution. By carefully structuring how information travels (locally first, then hierarchically), you can keep the "identity" of every user action alive until the very end.
- Robustness to "Inactive" Domains: The model performs significantly better than baselines for users who have very little data in certain domains. Because it doesn't force history into a compressed summary, it can better extract signal from the few sparse events that are present.
- Efficiency at Scale: The architecture scales much better than standard Transformers. When history length was doubled to 3,072 events, TokenFormer V2 remained efficient and outperformed baselines that had access to the same amount of data but used more compute.
Key Results
- Efficiency: On the TAAC 2026 benchmark, it achieved top-tier accuracy while using only 19.5% of the forward FLOPs required by MixFormer (a leading baseline).
- Accuracy: It delivered a +0.10 pp AUC gain over MixFormer on the benchmark and a +0.07 pp gain in billion-parameter industrial evaluations.
- Speed: Inference was measured at 2.6× faster than the strongest predictive baseline.
- Business Impact: In live A/B testing against a production model, it yielded a 2.33% increase in Gross Merchandise Value (GMV), a statistically significant and massive win in the context of large-scale advertising.
Limitations and Future Work
While the paper demonstrates excellent results, a few areas remain for further exploration:
- Branching Complexity: For production serving, the authors mention splitting history into branches with different depths. Managing these branches adds engineering complexity to the data pipeline.
- Hardware Specialization: The current gains are algorithmic (FLOPs and memory). The authors suggest that further gains could be found by optimizing the group-wise kernels for specific hardware (like H20 GPUs).
- Generalization: While tested on advertising, the "full-resolution" approach's effectiveness in other domains like generative search or long-form document processing wasn't explored.
Summary updated