TokenFormer V2: Breaking the Resolution-Complexity Dilemma in Unified Recommendation

Kaihui Wu, Guoming Li, Liang Guo, Yifeng Zhou, Yuehong Hu, Junwei Pan, Zhixiang Feng, Chengguo Yin, Wenqi Fan, Haijie Gu, Jie JiangarXiv 2026 · arXiv

TL;DR

TokenFormer V2 is a new unified recommendation architecture that solves the "resolution-complexity dilemma" by decoupling behavior detail from computational cost. By using a group-wise token layout and hierarchical merging, it preserves every individual behavior token throughout the model while achieving state-of-the-art accuracy with 5.1× fewer FLOPs than leading baselines and a 2.33% increase in GMV in live production.

What Problem Does This Paper Address?

Industrial recommendation systems face a frustrating trade-off. To predict what a user wants, you need their history. Long histories provide rich evidence, but modeling them with standard Transformers is computationally ruinous because self-attention cost grows quadratically with sequence length.

To stay within "serving budgets" (the time and compute allowed to generate a recommendation), engineers usually do one of two things:

  1. Truncate history: Throw away older data.
  2. Compress history: Use pooling or "summary tokens" to represent groups of actions.

Both approaches lose the "fine-grained evidence" found in specific, individual actions. TokenFormer V2 aims to provide full-resolution modeling—keeping every single behavior event accessible to the final prediction—without the massive computational overhead.

Methods

The core innovation is a local-global decomposition paradigm. Instead of letting every token talk to every other token (dense attention), the model organizes information flow through four key mechanisms:

  • Group-wise Token Layout: The behavior sequence is partitioned into contiguous segments. Each segment is bundled with non-sequential context (user/item features) and special "group relay" tokens.
  • Local Group Attention: In each layer, tokens only perform dense attention within their small group. This keeps the initial computation linear relative to the total history length.
  • Hierarchical Group Merging: As the data moves to deeper layers, adjacent groups merge. This gradually expands the "interaction range" (how far back in time a behavior can "see") without ever pooling or discarding the individual behavior tokens.
  • NS Fusion & Global Sync: These modules act as the "highways" for information. NS Fusion mixes non-sequential field features across the whole model, while Global Sync uses a small set of "readout tokens" to coordinate global context across all groups.
  • Full-Resolution Readout: Unlike models that predict based on a single summary vector, the final stage of TokenFormer V2 uses cross-attention to look back at every encoded behavior token in the sequence.

Datasets

The authors validated the model across three scales:

  1. TAAC 2026 Benchmark: The Tencent Advertising Algorithm Competition 2026 dataset, featuring 47.1 million impressions and histories from four distinct behavior domains.
  2. WeChat Channels (Offline): A massive industrial dataset used to train billion-parameter models on historical advertising logs.
  3. Live Online A/B Testing: Real-world traffic on WeChat Channels advertising to measure actual business impact (GMV).

Key Insights and Ideas

  • Resolution $\neq$ Density: The most important takeaway is that you don't need dense interaction to maintain high resolution. By carefully structuring how information travels (locally first, then hierarchically), you can keep the "identity" of every user action alive until the very end.
  • Robustness to "Inactive" Domains: The model performs significantly better than baselines for users who have very little data in certain domains. Because it doesn't force history into a compressed summary, it can better extract signal from the few sparse events that are present.
  • Efficiency at Scale: The architecture scales much better than standard Transformers. When history length was doubled to 3,072 events, TokenFormer V2 remained efficient and outperformed baselines that had access to the same amount of data but used more compute.

Key Results

  • Efficiency: On the TAAC 2026 benchmark, it achieved top-tier accuracy while using only 19.5% of the forward FLOPs required by MixFormer (a leading baseline).
  • Accuracy: It delivered a +0.10 pp AUC gain over MixFormer on the benchmark and a +0.07 pp gain in billion-parameter industrial evaluations.
  • Speed: Inference was measured at 2.6× faster than the strongest predictive baseline.
  • Business Impact: In live A/B testing against a production model, it yielded a 2.33% increase in Gross Merchandise Value (GMV), a statistically significant and massive win in the context of large-scale advertising.

Limitations and Future Work

While the paper demonstrates excellent results, a few areas remain for further exploration:

  • Branching Complexity: For production serving, the authors mention splitting history into branches with different depths. Managing these branches adds engineering complexity to the data pipeline.
  • Hardware Specialization: The current gains are algorithmic (FLOPs and memory). The authors suggest that further gains could be found by optimizing the group-wise kernels for specific hardware (like H20 GPUs).
  • Generalization: While tested on advertising, the "full-resolution" approach's effectiveness in other domains like generative search or long-form document processing wasn't explored.

Summary updated

Abstract

Industrial recommendation faces a resolution--complexity dilemma: dense interactions make detailed modeling of long behavior histories expensive, while compact history summaries restrict the evidence available to prediction. A unified backbone must preserve rich behavioral details within strict serving budgets. We introduce \textbf{TokenFormer V2}, a unified architecture that explicitly decouples \emph{representation resolution} from \emph{interaction density}. Its group-wise token layout jointly encodes behavior segments with non-sequential context, expands interaction ranges via hierarchical group merging, and coordinates distributed groups through \emph{NS Fusion} and \emph{Global Sync} modules. Through this \emph{local--global decomposition} paradigm, computation is concentrated on structured interactions while preserving fine-grained evidence, ensuring that every encoded event remains directly accessible to the \emph{Full-Resolution Readout} module. On the Tencent Advertising Algorithm Competition 2026 (TAAC 2026) benchmark, TokenFormer V2 achieves top-ranking accuracy among unified recommendation baselines while using \textbf{5.1$\times$ fewer FLOPs} than leading baselines, sustaining its advantage across long histories and imbalanced behavior domains. Furthermore, in billion-parameter evaluations on WeChat Channels advertising, it consistently outperforms state-of-the-art industrial baselines, yielding up to a \textbf{+0.07 pp} overall AUC gain. In live online A/B testing against the production model, TokenFormer V2 delivers a statistically significant \textbf{2.33\% increase in GMV}. These results establish TokenFormer V2 as a battle-tested blueprint for scalable, full-resolution behavior modeling in industrial recommendation.
BibTeX
@misc{wu2026tokenformer,
  author = {Kaihui Wu and Guoming Li and Liang Guo and Yifeng Zhou and Yuehong Hu and Junwei Pan and Zhixiang Feng and Chengguo Yin and Wenqi Fan and Haijie Gu and Jie Jiang},
  title = {{TokenFormer V2: Breaking the Resolution-Complexity Dilemma in Unified Recommendation}},
  year = {2026},
  eprint = {2611.19074},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19074},
}
Figure