Abstract
Click-Through Rate (CTR) prediction is a cornerstone of digital advertising and recommendation systems, typically characterized by high-dimensional, sparse categorical data. While normalization techniques like Batch Normalization (BN) and Layer Normalization (LN) are standard in computer vision and natural language processing, their systematic application and impact on CTR models have been less understood. This paper presents a comprehensive study of normalization in CTR prediction, introducing Variance-Only LayerNorm (VO-LN) and the NormDNN architecture. We find that variance scaling is the primary driver of performance improvements, enabling faster learning from sparse features and acting as a noise filter. NormDNN achieves superior performance on industry benchmarks like Criteo and Avazu, outperforming more complex models with significantly lower computational overhead.
Introduction
Modern CTR prediction models often follow an "Embedding + MLP" architecture to handle a mix of sparse categorical and dense numerical features. Despite the ubiquity of normalization in deep learning, its use in CTR has been inconsistent. This research addresses the fundamental questions: which normalization method is most effective for CTR data, where should it be applied, and what are the underlying mechanisms of its success?
Methodology
The authors propose two primary innovations:
- Variance-Only LayerNorm (VO-LN): Standard LayerNorm involves mean-centering, variance scaling, and learnable gain/bias parameters. The researchers discovered that for CTR tasks, mean-centering and learnable parameters are often redundant or detrimental. VO-LN simplifies the operation by only scaling the input by its variance.
- NormDNN Architecture: A specific configuration for applying normalization:
- Numerical Features: VO-LN or standard LN applied to embeddings.
- Categorical Features: BatchNorm applied to embeddings.
- MLP Layers: VO-LN applied before the ReLU activation function.
The Impact of Normalization on Feature Distributions
A key finding of this study is that the benefits of normalization in CTR models stem almost entirely from variance scaling. By normalizing the variance, the model becomes more sensitive to the long-tail, low-frequency features common in sparse datasets.
As shown in Figure 1, applying VO-LN significantly alters the distribution of activations. Notably, applying VO-LN to MLP layers (blue bars) shifts the mean activation (mu) deep into the negative range, which is critical for the subsequent activation function.
The "Noise Filter" Effect
This negative shift in the mean activation has a profound effect when combined with ReLU activations. Since ReLU zeroes out negative inputs, a large portion of the neuron responses are effectively silenced.
Figure 3 illustrates this distribution shift. By pushing many activations into the negative range, VO-LN acts as a "noise filter," preventing weak or irrelevant signals from interfering with the training process. This mechanism allows the model to focus on the most informative features, leading to better generalization and faster convergence.
Experimental Results
The proposed NormDNN was evaluated on three large-scale datasets: Criteo (45M instances), Malware (8.92M instances), and Avazu (40.43M instances).
- Criteo: NormDNN achieved an AUC of 0.8107, significantly higher than the standard DNN (0.8054) and even the more complex xDeepFM (0.8063).
- Universality: Integrating the proposed normalization strategy into existing models like DeepFM and xDeepFM consistently improved their performance, proving the robustness of the approach.
- Efficiency: Unlike models that rely on complex feature interaction layers, NormDNN maintains the efficiency of a standard MLP, making it highly suitable for high-throughput production environments.
Conclusion
Correct normalization is a critical, yet often overlooked, component of high-performing CTR models. By focusing on variance scaling through VO-LN and following the NormDNN placement recipe, practitioners can achieve state-of-the-art results with minimal computational cost. Future work may explore adaptive normalization strategies tailored to varying levels of feature sparsity and different activation functions.
Summary updated