Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction

Zhiqiang Wang, Qingyun She, Pengtao Zhang, Junlin ZhangarXiv 2020 · arXiv

Abstract

Click-Through Rate (CTR) prediction is a cornerstone of digital advertising and recommendation systems, typically characterized by high-dimensional, sparse categorical data. While normalization techniques like Batch Normalization (BN) and Layer Normalization (LN) are standard in computer vision and natural language processing, their systematic application and impact on CTR models have been less understood. This paper presents a comprehensive study of normalization in CTR prediction, introducing Variance-Only LayerNorm (VO-LN) and the NormDNN architecture. We find that variance scaling is the primary driver of performance improvements, enabling faster learning from sparse features and acting as a noise filter. NormDNN achieves superior performance on industry benchmarks like Criteo and Avazu, outperforming more complex models with significantly lower computational overhead.

Introduction

Modern CTR prediction models often follow an "Embedding + MLP" architecture to handle a mix of sparse categorical and dense numerical features. Despite the ubiquity of normalization in deep learning, its use in CTR has been inconsistent. This research addresses the fundamental questions: which normalization method is most effective for CTR data, where should it be applied, and what are the underlying mechanisms of its success?

Methodology

The authors propose two primary innovations:

  1. Variance-Only LayerNorm (VO-LN): Standard LayerNorm involves mean-centering, variance scaling, and learnable gain/bias parameters. The researchers discovered that for CTR tasks, mean-centering and learnable parameters are often redundant or detrimental. VO-LN simplifies the operation by only scaling the input by its variance.
  2. NormDNN Architecture: A specific configuration for applying normalization:
  3. Numerical Features: VO-LN or standard LN applied to embeddings.
  4. Categorical Features: BatchNorm applied to embeddings.
  5. MLP Layers: VO-LN applied before the ReLU activation function.

The Impact of Normalization on Feature Distributions

A key finding of this study is that the benefits of normalization in CTR models stem almost entirely from variance scaling. By normalizing the variance, the model becomes more sensitive to the long-tail, low-frequency features common in sparse datasets.

Figure 1: Comparison of mean (mu) and standard deviation (sigma) across different model components and normalization strategies.

As shown in Figure 1, applying VO-LN significantly alters the distribution of activations. Notably, applying VO-LN to MLP layers (blue bars) shifts the mean activation (mu) deep into the negative range, which is critical for the subsequent activation function.

The "Noise Filter" Effect

This negative shift in the mean activation has a profound effect when combined with ReLU activations. Since ReLU zeroes out negative inputs, a large portion of the neuron responses are effectively silenced.

Figure 3: Distribution of mu and sigma for standard DNN vs. EMB+VO-LN, illustrating the shift in activation patterns.

Figure 3 illustrates this distribution shift. By pushing many activations into the negative range, VO-LN acts as a "noise filter," preventing weak or irrelevant signals from interfering with the training process. This mechanism allows the model to focus on the most informative features, leading to better generalization and faster convergence.

Experimental Results

The proposed NormDNN was evaluated on three large-scale datasets: Criteo (45M instances), Malware (8.92M instances), and Avazu (40.43M instances).

  • Criteo: NormDNN achieved an AUC of 0.8107, significantly higher than the standard DNN (0.8054) and even the more complex xDeepFM (0.8063).
  • Universality: Integrating the proposed normalization strategy into existing models like DeepFM and xDeepFM consistently improved their performance, proving the robustness of the approach.
  • Efficiency: Unlike models that rely on complex feature interaction layers, NormDNN maintains the efficiency of a standard MLP, making it highly suitable for high-throughput production environments.

Conclusion

Correct normalization is a critical, yet often overlooked, component of high-performing CTR models. By focusing on variance scaling through VO-LN and following the NormDNN placement recipe, practitioners can achieve state-of-the-art results with minimal computational cost. Future work may explore adaptive normalization strategies tailored to varying levels of feature sparsity and different activation functions.

Summary updated

BibTeX
@misc{wang2020correct,
  author = {Zhiqiang Wang and Qingyun She and Pengtao Zhang and Junlin Zhang},
  title = {{Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction}},
  year = {2020},
  eprint = {2006.12753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.12753},
}
Figure