DNA: Unlocking Depth Scaling in CTR Models

Caiyuan Li, Wenjie Wu, Yuheng Huang, Xiao Zhou, Silu Zhou, Ke Chen, Zinan Zhao, Xinrui Zeng, Hanmo Xu, Chaoqun Song, Hua Zong, Zongyuan Wu, Hongbin Zhou, Huimin Yi, Zhixuan Zhang, Dan Ou, Haihong TangarXiv 2026 · arXiv

In the high-stakes world of online advertising and search, predicting Click-Through Rate (CTR) is the engine that drives revenue. Modern models rely heavily on "long user behavior sequences"—the digital breadcrumbs of everything a user has clicked or viewed. While cross-attention has made processing these massive sequences feasible, researchers at Alibaba have identified a frustrating plateau: simply stacking more layers doesn't seem to help much anymore.

Their new paper, "DNA: Unlocking Depth Scaling in CTR Models," introduces a novel architecture that breaks through this bottleneck, allowing deeper models to finally deliver on their promise.

TL;DR

The authors identify that the "static" nature of behavior sequences in standard cross-attention models prevents them from benefiting from increased depth. They propose Dual-stream Nexus Attention (DNA), which allows the behavior sequence to evolve layer-by-layer based on the current search request, leading to significant gains in accuracy and real-world revenue at Taobao.

What Problem Does This Paper Address?

Industrial search rankers must score thousands of candidate items against a user's long history (often 10,000+ behaviors) within milliseconds. The standard tool for this is cross-attention, where a "request" (query + user features) "reads" the behavior sequence.

However, the authors observed a "depth-scaling bottleneck." On Taobao search, increasing a cross-attention stack from 16 to 28 layers yielded a measly 0.02 percentage point (pt) gain in AUC (Area Under the Curve).

The culprit? In standard cross-attention, the behavior sequence is projected into keys and values once per layer, but the sequence itself never changes. Adding depth just means the model reads the same fixed history over and over. To get more out of depth, the sequence itself needs to "evolve" as the model processes the request.

Methods

The core innovation is DNA (Dual-stream Nexus Attention). It moves away from a one-way "read" to a two-way "dialogue" between the request and the behavior history.

  1. Dual Streams: The model maintains two parallel representations: the Request Stream (query, user, scene) and the Candidate Stream (features of a specific item).
  2. The Read-Update Cycle: In every layer:
  3. Read: Both the request and candidate streams use cross-attention to "read" the current behavior sequence.
  4. Update: Crucially, the behavior sequence then "reads" the updated request stream. Each behavior token attends to the request tokens and updates its own representation.
  5. Efficiency through Sharing: Updating a 10,000-item sequence for every candidate would be computationally suicidal. DNA solves this by making the sequence update candidate-independent. The sequence is updated once per request, and that updated state is shared across all thousands of candidates being ranked.
Figure 2: Schematic information flow in long-sequence ranking. Rows receive information from columns. S, U and T denote the behavior sequence, request features and features specific to one candidate. The full non-sequential representation F uses both request and candidate features. Blue, purple, teal and orange mark attention involving S, and gray marks interactions among non-sequential features. The same colors are used in all figures: teal for the request stream and DNA, orange for candidate features and DNA-T. R1 asks whether the sequence incorporates features of the current request. R2 asks whether candidates within a request can share the same sequence state. R3 asks whether attention cost grows linearly with sequence length 𝐿at fixed non-sequential token counts and model width.

Datasets

The models were trained and evaluated using:

  • Taobao Search Impression Logs: Massive industrial datasets containing multiple user behavior sequences.
  • Online A/B Testing: Real-world deployment on Taobao's production search engine.

Key Insights and Ideas

  • The KV Bottleneck: The "aha moment" was realizing that the bottleneck wasn't in the queries, but in the static Keys and Values (KV) of the behavior sequence. By letting the sequence evolve, the model can perform more sophisticated reasoning over time.
  • Contextual Evolution: Because the sequence updates are conditioned on the request, the same history is "interpreted" differently depending on whether a user is searching for "running shoes" versus "water bottles."
  • Linear Complexity: Despite the added complexity of the update, DNA maintains linear compute complexity relative to sequence length, making it practical for production.

Key Results

The results show that DNA successfully "unlocks" the benefits of deeper models:

  • Superior Scaling: As shown in Figure 1, DNA continues to improve as it gets deeper, whereas the baseline flattens out. A 10-layer DNA model significantly outperforms a 28-layer baseline.
  • Efficiency: DNA-4 (4 layers) exceeds the performance of a 28-layer standard stack while using only about 40% of the FLOPs (Floating Point Operations).
  • Offline Gains: DNA improved AUC by 0.08 to 0.26 pt across various compute budgets compared to standard cross-attention.
  • Production Impact: In a live A/B test on Taobao, DNA-4 improved CTR by 0.8% and Gross Merchandise Volume (GMV) by 0.5%, leading to its full deployment.
Figure 1: Depth scaling on Taobao search. Over the same com- pute increase (shaded), growing the cross-attention baseline from 16 to 28 layers adds 0.02 pt of AUC, while growing DNA from 6 to 10 layers adds 0.13 pt. Each marker is one model, labeled with its number of layers. ΔAUC is relative to the four-layer baseline, CrossAttn-4.

Limitations and Future Work

While DNA is highly effective, the authors note some areas for exploration:

  • Candidate-Aware Updates: They tested a version where the sequence also updates based on the specific candidate item (DNA-T). While slightly more accurate in some cases, it was too expensive for serving because the sequence couldn't be shared.
  • Extreme Depth: The study went up to 10 layers for DNA; exploring even deeper architectures or longer histories (e.g., multi-year) remains a future goal.
  • Generative Ranking: The authors plan to extend this "evolving sequence" approach to the emerging field of generative ranking.

In summary, DNA provides a blueprint for making deep learning models actually "deeper" in the context of recommendation systems, proving that sometimes, the best way to understand the past is to let the present talk back to it.

Summary updated

Abstract

Long user behavior sequences are among the strongest signals for click-through rate (CTR) prediction. Cross-attention makes these sequences practical for industrial ranking by keeping compute linear in sequence length. Yet the gains from stacking cross-attention layers taper off: on Taobao search, growing a cross-attention stack from 16 to 28 layers adds only 0.02 percentage points (pt) of AUC. We argue that the key-value side is the bottleneck. Each layer projects the same behavior sequence into keys and values, so adding depth gives the model more reads of a fixed sequence. We propose DNA (Dual-stream Nexus Attention), which lets the sequence itself evolve with depth. At each layer, the request representation (the search query, user features and scene) first reads the sequence. Each behavior token then attends to the updated request representation and uses what it reads to update itself, so the next layer reads a sequence shaped by the current request. The update runs once per request, all candidates share the resulting sequence, and both the reads and the update keep linear complexity in sequence length. At comparable forward FLOPs, DNA improves AUC over cross-attention stacks by 0.08 to 0.26 pt and keeps improving in the depth range where those stacks level off. DNA-4 is deployed in Taobao search. In an online A/B test against the production model, it improves CTR by 0.8% and GMV by 0.5%.
BibTeX
@misc{li2026dna,
  author = {Caiyuan Li and Wenjie Wu and Yuheng Huang and Xiao Zhou and Silu Zhou and Ke Chen and Zinan Zhao and Xinrui Zeng and Hanmo Xu and Chaoqun Song and Hua Zong and Zongyuan Wu and Hongbin Zhou and Huimin Yi and Zhixuan Zhang and Dan Ou and Haihong Tang},
  title = {{DNA: Unlocking Depth Scaling in CTR Models}},
  year = {2026},
  eprint = {2611.25659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25659},
}
Figure