In the high-stakes world of online advertising and search, predicting Click-Through Rate (CTR) is the engine that drives revenue. Modern models rely heavily on "long user behavior sequences"—the digital breadcrumbs of everything a user has clicked or viewed. While cross-attention has made processing these massive sequences feasible, researchers at Alibaba have identified a frustrating plateau: simply stacking more layers doesn't seem to help much anymore.
Their new paper, "DNA: Unlocking Depth Scaling in CTR Models," introduces a novel architecture that breaks through this bottleneck, allowing deeper models to finally deliver on their promise.
TL;DR
The authors identify that the "static" nature of behavior sequences in standard cross-attention models prevents them from benefiting from increased depth. They propose Dual-stream Nexus Attention (DNA), which allows the behavior sequence to evolve layer-by-layer based on the current search request, leading to significant gains in accuracy and real-world revenue at Taobao.
What Problem Does This Paper Address?
Industrial search rankers must score thousands of candidate items against a user's long history (often 10,000+ behaviors) within milliseconds. The standard tool for this is cross-attention, where a "request" (query + user features) "reads" the behavior sequence.
However, the authors observed a "depth-scaling bottleneck." On Taobao search, increasing a cross-attention stack from 16 to 28 layers yielded a measly 0.02 percentage point (pt) gain in AUC (Area Under the Curve).
The culprit? In standard cross-attention, the behavior sequence is projected into keys and values once per layer, but the sequence itself never changes. Adding depth just means the model reads the same fixed history over and over. To get more out of depth, the sequence itself needs to "evolve" as the model processes the request.
Methods
The core innovation is DNA (Dual-stream Nexus Attention). It moves away from a one-way "read" to a two-way "dialogue" between the request and the behavior history.
- Dual Streams: The model maintains two parallel representations: the Request Stream (query, user, scene) and the Candidate Stream (features of a specific item).
- The Read-Update Cycle: In every layer:
- Read: Both the request and candidate streams use cross-attention to "read" the current behavior sequence.
- Update: Crucially, the behavior sequence then "reads" the updated request stream. Each behavior token attends to the request tokens and updates its own representation.
- Efficiency through Sharing: Updating a 10,000-item sequence for every candidate would be computationally suicidal. DNA solves this by making the sequence update candidate-independent. The sequence is updated once per request, and that updated state is shared across all thousands of candidates being ranked.
Datasets
The models were trained and evaluated using:
- Taobao Search Impression Logs: Massive industrial datasets containing multiple user behavior sequences.
- Online A/B Testing: Real-world deployment on Taobao's production search engine.
Key Insights and Ideas
- The KV Bottleneck: The "aha moment" was realizing that the bottleneck wasn't in the queries, but in the static Keys and Values (KV) of the behavior sequence. By letting the sequence evolve, the model can perform more sophisticated reasoning over time.
- Contextual Evolution: Because the sequence updates are conditioned on the request, the same history is "interpreted" differently depending on whether a user is searching for "running shoes" versus "water bottles."
- Linear Complexity: Despite the added complexity of the update, DNA maintains linear compute complexity relative to sequence length, making it practical for production.
Key Results
The results show that DNA successfully "unlocks" the benefits of deeper models:
- Superior Scaling: As shown in Figure 1, DNA continues to improve as it gets deeper, whereas the baseline flattens out. A 10-layer DNA model significantly outperforms a 28-layer baseline.
- Efficiency: DNA-4 (4 layers) exceeds the performance of a 28-layer standard stack while using only about 40% of the FLOPs (Floating Point Operations).
- Offline Gains: DNA improved AUC by 0.08 to 0.26 pt across various compute budgets compared to standard cross-attention.
- Production Impact: In a live A/B test on Taobao, DNA-4 improved CTR by 0.8% and Gross Merchandise Volume (GMV) by 0.5%, leading to its full deployment.
Limitations and Future Work
While DNA is highly effective, the authors note some areas for exploration:
- Candidate-Aware Updates: They tested a version where the sequence also updates based on the specific candidate item (DNA-T). While slightly more accurate in some cases, it was too expensive for serving because the sequence couldn't be shared.
- Extreme Depth: The study went up to 10 layers for DNA; exploring even deeper architectures or longer histories (e.g., multi-year) remains a future goal.
- Generative Ranking: The authors plan to extend this "evolving sequence" approach to the emerging field of generative ranking.
In summary, DNA provides a blueprint for making deep learning models actually "deeper" in the context of recommendation systems, proving that sometimes, the best way to understand the past is to let the present talk back to it.
Summary updated