Abstract
Normalization is a cornerstone of modern deep learning, providing the stability necessary to train deep architectures. However, traditional methods like Group Normalization (GN) rely on fixed, predefined channel groups, which can lead to suboptimal performance when disparate feature statistics are merged. We introduce Dynamic Group Normalization (DGN), a framework that replaces static grouping with a flexible, data-driven reassignment process. By periodically clustering channels based on their statistical signatures (mean and variance), DGN adapts to evolving data distributions throughout the training process. Our extensive evaluation across 34 architectures and 11 benchmarks demonstrates that DGN consistently outperforms standard normalization techniques, particularly in challenging scenarios involving long-tailed data and out-of-distribution generalization.
Introduction
Normalization layers are often described as the "secret sauce" of deep neural networks, stabilizing internal data distributions to facilitate faster convergence and better performance. Despite their success, existing methods suffer from a fundamental "rigidity" problem. Batch Normalization (BN) is highly dependent on batch size, while batch-independent alternatives like Group Normalization (GN) or Layer Normalization (LN) rely on fixed, architectural groupings of channels.
The authors argue that these fixed groups are suboptimal. For instance, if a high-variance channel (detecting complex textures) is grouped with a low-variance channel (detecting simple edges), the resulting group statistics "dilute" the unique features of both. This lack of flexibility is particularly detrimental when dealing with "long-tail" data distributions or "out-of-distribution" (OOD) scenarios, where the model must adapt to statistics that differ significantly from the training average.
The DGN Framework
Dynamic Group Normalization (DGN) treats channel grouping as an evolving process rather than a static design choice. The mechanism operates through two distinct but complementary phases:
1. Offline Learning Phase (Regrouping)
Periodically during training (e.g., every 20 epochs), the model evaluates the statistical signature of every channel. It identifies groups that have become too heterogeneous and employs a constrained K-Means clustering algorithm to reassign channels into more statistically coherent groups. This ensures that channels with similar means and variances are normalized together, preserving their discriminative power.
2. Online Learning Phase (Normalizing)
During standard training iterations, the model performs group-wise normalization using the most recently optimized group assignments. To maintain computational efficiency, DGN keeps channels in their original physical positions and utilizes a reference index for grouping, avoiding the overhead of physical memory movement.
To ensure training stability, DGN follows a Dynamic Grouping Schedule. Regrouping begins after an initial warm-up period and eventually "freezes" toward the end of training, allowing the model to converge to a stable state for inference.
Experimental Evaluation
The authors conducted an exhaustive evaluation using 34 different architectures across 11 benchmarks, covering classification, object detection, and semantic segmentation.
Key Results
- Standard Classification: On ImageNet, DGN achieved a +1.12% accuracy boost for MobileNetV3 and +0.98% for CAFormer architectures compared to standard GN.
- Object Detection: DGN improved YOLOv9-C performance by 2.01% (mAP@50:95) over the baseline.
- Long-Tail Distributions: DGN showed its greatest strength here, improving ViT-Base with LIFT by 2.66% on ImageNet-LT.
- Robustness and OOD: DGN demonstrated significantly lower error rates on corrupted data (ImageNet-C) and superior AUROC scores for out-of-distribution detection, proving its ability to handle data shifts better than static methods.
Conclusion and Future Work
DGN demonstrates that statistical awareness in normalization is key to model robustness. By allowing groups to adapt spatially across layers and temporally across epochs, DGN preserves feature integrity without significant computational cost—adding less than 1% runtime overhead.
Future research may explore test-time dynamic adaptation to further boost performance in rapidly changing environments. Additionally, while this work focuses on computer vision, the principles of DGN are highly applicable to other domains like Large Language Models (LLMs) and time-series analysis, where data statistics are known to shift dramatically.
Summary updated