How Does BN Increase Collapsed Neural Network Filters?

Sheng Zhou, Xinjiang Wang, Ping Luo, Litong Feng, Wenjie Li, Wei ZhangarXiv 2020 · OpenReview

Abstract

In deep learning, structured sparsity is often sought for model compression. However, recent research identifies a phenomenon termed "filter collapse," where the combination of Batch Normalization (BN) and ReLU activations spontaneously deactivates neural network filters during training. This unintended sparsity reduces representational capacity and degrades performance. This paper analyzes the mathematical mechanism behind this collapse—a "frozen zone" in parameter space—and proposes "post-shifted BN" (psBN), a simple modification that reactivates dead filters and improves accuracy in classification and detection tasks.

Introduction

Standard deep learning architectures like ResNet and VGG-BN naturally develop structured sparsity, where entire convolution kernels and their associated BN parameters effectively drop to zero. While sparsity is typically viewed as a benefit for efficiency, this specific type of collapse is a form of capacity loss rather than optimized compression. It occurs most frequently during "warm restarts" (cyclic learning rates), fine-tuning, or when using large learning rates. The authors investigate the optimization landscape to explain why filters become trapped in a non-trainable state and how to prevent this degradation.

The BN-ReLU Trap and Filter Collapse

The researchers identified that the combination of BN and ReLU creates a "frozen zone" in the parameter space. When a filter's parameters fall into this region, the gradient becomes zero. At this point, standard weight decay acts as a one-way street, pulling parameters toward absolute zero with no gradient signal to pull them back out. High gradient noise—driven by large learning rates or small batch sizes—acts as the catalyst, knocking filters into this trap.

Figure 1: (a) Distribution of filter L1 norms showing a significant spike at zero for VGG with BN. (b) Trajectories of BN parameters γ and β being sucked into the origin (zero), representing the "frozen zone."

Through analytical modeling, the authors proved that the probability of a filter becoming sparse is proportional to the square of the learning rate and inversely proportional to the square of the BN scale parameter ($\gamma$). Experimental diagnostics tracking the "trace" of BN parameters ($\gamma$ and $\beta$) visualize how they converge to the origin and remain stuck, as shown in Figure 1.

Proposed Solution: Post-Shifted BN (psBN)

To solve the issue, the authors proposed a modified BN layer called post-shifted BN (psBN):

$$y = BN(\hat{x}) + \alpha$$

By adding a small positive constant ($\alpha$, typically 0.1) after normalization but before the ReLU activation, the "frozen zone" is mathematically shifted. This shift makes the zero-state unstable under weight decay, ensuring that even if a filter is currently "dead," it remains in a region where it can still receive gradients. This allows the optimizer to reactivate collapsed filters later in training.

Key Results

The effectiveness of psBN was validated across CIFAR-10 and MS-COCO 2017 datasets. A critical observation was made during Snapshot Ensemble training, which utilizes cyclic learning rates. In standard BN, sparsity steadily climbs across training rounds, eventually leading to a total failure of the model to learn in later stages.

Figure 5: (a) Top-1 Accuracy vs. Iterations across five rounds of training, showing a total failure to learn by Round 5. (b) The corresponding increase in the sparsity ratio over training rounds.

As illustrated in Figure 5, the increasing sparsity ratio directly correlates with the degradation of training accuracy. By implementing psBN, the researchers were able to prevent this climb in sparsity. For ResNet-110 on CIFAR-10, psBN achieved 94.86% accuracy compared to the 94.63% baseline. In object detection tasks on MS-COCO, using psBN in a RetinaNet model led to an increase in Mean Average Precision (mAP) from 35.8 to 36.4.

Conclusion

Filter collapse is a significant but often overlooked bug in the optimization of BN-ReLU networks. The proposed psBN provides a robust "escape hatch" for dead filters by changing the optimization landscape without altering the network's representational potential. While this study focuses on BN and ReLU, future work may explore similar dynamics in modern components like LayerNorm, Swish, or GELU to determine if they suffer from analogous traps.

Summary updated

Abstract

Improving sparsity of deep neural networks (DNNs) is essential for network compression and has drawn much attention. In this work, we disclose a harmful sparsifying process called filter collapse, which is common in DNNs with batch normalization (BN) and rectified linear activation functions (e.g. ReLU, Leaky ReLU). It occurs even without explicit sparsity-inducing regularizations such as $L_1$. This phenomenon is caused by the normalization effect of BN, which induces a non-trainable region in the parameter space and reduces the network capacity as a result. This phenomenon becomes more prominent when the network is trained with large learning rates (LR) or adaptive LR schedulers, and when the network is finetuned. We analytically prove that the parameters of BN tend to become sparser during SGD updates with high gradient noise and that the sparsifying probability is proportional to the square of learning rate and inversely proportional to the square of the scale parameter of BN. To prevent the undesirable collapsed filters, we propose a simple yet effective approach named post-shifted BN (psBN), which has the same representation ability as BN while being able to automatically make BN parameters trainable again as they saturate during training. With psBN, we can recover collapsed filters and increase the model performance in various tasks such as classification on CIFAR-10 and object detection on MS-COCO2017.
BibTeX
@misc{zhou2020how,
  author = {Sheng Zhou and Xinjiang Wang and Ping Luo and Litong Feng and Wenjie Li and Wei Zhang},
  title = {{How Does BN Increase Collapsed Neural Network Filters?}},
  year = {2020},
  eprint = {2001.11216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.11216},
}
Figure