Abstract
In deep learning, structured sparsity is often sought for model compression. However, recent research identifies a phenomenon termed "filter collapse," where the combination of Batch Normalization (BN) and ReLU activations spontaneously deactivates neural network filters during training. This unintended sparsity reduces representational capacity and degrades performance. This paper analyzes the mathematical mechanism behind this collapse—a "frozen zone" in parameter space—and proposes "post-shifted BN" (psBN), a simple modification that reactivates dead filters and improves accuracy in classification and detection tasks.
Introduction
Standard deep learning architectures like ResNet and VGG-BN naturally develop structured sparsity, where entire convolution kernels and their associated BN parameters effectively drop to zero. While sparsity is typically viewed as a benefit for efficiency, this specific type of collapse is a form of capacity loss rather than optimized compression. It occurs most frequently during "warm restarts" (cyclic learning rates), fine-tuning, or when using large learning rates. The authors investigate the optimization landscape to explain why filters become trapped in a non-trainable state and how to prevent this degradation.
The BN-ReLU Trap and Filter Collapse
The researchers identified that the combination of BN and ReLU creates a "frozen zone" in the parameter space. When a filter's parameters fall into this region, the gradient becomes zero. At this point, standard weight decay acts as a one-way street, pulling parameters toward absolute zero with no gradient signal to pull them back out. High gradient noise—driven by large learning rates or small batch sizes—acts as the catalyst, knocking filters into this trap.
Through analytical modeling, the authors proved that the probability of a filter becoming sparse is proportional to the square of the learning rate and inversely proportional to the square of the BN scale parameter ($\gamma$). Experimental diagnostics tracking the "trace" of BN parameters ($\gamma$ and $\beta$) visualize how they converge to the origin and remain stuck, as shown in Figure 1.
Proposed Solution: Post-Shifted BN (psBN)
To solve the issue, the authors proposed a modified BN layer called post-shifted BN (psBN):
$$y = BN(\hat{x}) + \alpha$$
By adding a small positive constant ($\alpha$, typically 0.1) after normalization but before the ReLU activation, the "frozen zone" is mathematically shifted. This shift makes the zero-state unstable under weight decay, ensuring that even if a filter is currently "dead," it remains in a region where it can still receive gradients. This allows the optimizer to reactivate collapsed filters later in training.
Key Results
The effectiveness of psBN was validated across CIFAR-10 and MS-COCO 2017 datasets. A critical observation was made during Snapshot Ensemble training, which utilizes cyclic learning rates. In standard BN, sparsity steadily climbs across training rounds, eventually leading to a total failure of the model to learn in later stages.
As illustrated in Figure 5, the increasing sparsity ratio directly correlates with the degradation of training accuracy. By implementing psBN, the researchers were able to prevent this climb in sparsity. For ResNet-110 on CIFAR-10, psBN achieved 94.86% accuracy compared to the 94.63% baseline. In object detection tasks on MS-COCO, using psBN in a RetinaNet model led to an increase in Mean Average Precision (mAP) from 35.8 to 36.4.
Conclusion
Filter collapse is a significant but often overlooked bug in the optimization of BN-ReLU networks. The proposed psBN provides a robust "escape hatch" for dead filters by changing the optimization landscape without altering the network's representational potential. While this study focuses on BN and ReLU, future work may explore similar dynamics in modern components like LayerNorm, Swish, or GELU to determine if they suffer from analogous traps.
Summary updated