Abstract
In light of the smoothness property brought by skip connections in ResNet, this paper proposed the Skip Logit to introduce the skip connection mechanism that fits arbitrary DNN dimensions and embraces similar properties to ResNet. Meta Tanh Normalization (MTN) is designed to learn variance information and stabilize the training process. With these delicate designs, our Skip Meta Logit (SML) brought incremental boosts to the performance of extensive SOTA ctr prediction models on two real-world datasets. In the meantime, we prove that the optimization landscape of arbitrarily deep skip logit networks has no spurious local optima. Finally, SML can be easily added to building blocks and has delivered offline accuracy and online business metrics gains on app ads learning to rank systems at TikTok.
Introduction
In the domain of recommendation systems, Click-Through Rate (CTR) prediction is the foundational task. While Deep Neural Networks (DNNs) are the standard building blocks, they face two significant hurdles in industrial settings: structural rigidity and online learning constraints. Popular techniques like ResNet require uniform layer widths, which are incompatible with the "tower" structures common in CTR models where layers progressively narrow. Furthermore, standard Batch Normalization (BN) often fails in online learning environments due to inconsistent batch sizes, leading to unstable statistics and poor convergence. Without stabilization, deep CTR models suffer from "rank collapse" and "dead neurons," preventing them from benefiting from increased depth.
Methodology
The Skip Meta Logit (SML) framework introduces two primary components to address these challenges:
- Skip Logit: Instead of sequential layer-to-layer connections, Skip Logit connects the output of every hidden layer directly to the final output logit. This "flattens" the residual structure, allowing gradients to flow directly from the loss function to shallow layers. This ensures shallow layers remain expressive and leads to stable linear variance growth rather than unstable exponential growth as depth increases.
- Meta Tanh Normalization (MTN): To replace BN in online learning, MTN utilizes a Tanh activation to cap outliers and a Meta-Learner (a small auxiliary network) to dynamically predict scaling factors for each layer's variance. This maintains network smoothness even when processing noisy, real-world data streams.
Experimental Setup
The researchers evaluated SML on two public benchmarks and one industrial system:
- Criteo: 45 million samples with 13 continuous and 26 categorical features.
- Avazu: 40 million samples across 23 fields.
- Industrial App Ads System: Real-world traffic from TikTok/ByteDance to measure online business metrics.
Analysis and Key Insights
Theoretical analysis demonstrates that Skip Logit networks have no spurious local optima under certain linear conditions, resulting in a smoother loss landscape. Empirically, SML effectively mitigates rank collapse—a phenomenon where deep layers lose the ability to distinguish between different inputs. By maintaining information flow from shallow layers, SML allows models to remain informative at extreme depths. While standard vanilla DNNs typically collapse and fail to train effectively beyond 27 layers, SML-equipped networks can be trained stably up to 100 layers.
Results
SML provided universal improvements when integrated into State-of-the-Art (SOTA) models, including DeepFM, xDeepFM, AutoInt, and DCN-V2. Notably, DCN-V2 combined with SML achieved a new best-in-class performance on the Criteo dataset.
In real-world A/B testing, SML delivered a +0.15% AUC lift and a +1.13% to +3.28% increase in Advertiser Value (ADVV). These margins are considered highly significant in high-stakes digital advertising systems.
Conclusion
SML addresses the fundamental stability issues of deep CTR models by introducing a flexible skip-connection mechanism and meta-learning-based normalization. While the current theoretical guarantees are strongest for linear cases, empirical results confirm the framework's efficacy in non-linear industrial applications. Future work will focus on refining the meta-normalization module to incorporate more sophisticated online statistical information.
Summary updated