TL;DR
This paper introduces Fine-grained Batch Normalization (FBN), a novel normalization technique that normalizes intermediate summation states within convolutional layers. This allows for the creation of "Finet" networks, which achieve state-of-the-art accuracy on ImageNet with competitive computational costs and significantly improved inference speed due to efficient fusion with convolutional layers.
What Problem Does This Paper Address?
The rapid advancement of deep neural networks has led to increasingly complex and resource-intensive models. While these models achieve impressive performance, their computational demands hinder their deployment on resource-constrained devices like mobile phones and embedded systems. This has spurred significant research into "light-weight" neural networks, aiming to reduce computational complexity and parameter count without sacrificing accuracy. However, existing methods often face trade-offs between training efficiency, inference speed, and accuracy. This paper addresses the challenge of designing light-weight networks that are both efficient during training and highly performant during inference, particularly on mobile platforms.
Methods
The core innovation is Fine-grained Batch Normalization (FBN). Unlike standard Batch Normalization (BN) which normalizes the final output of a weighted sum, FBN normalizes the intermediate summation states. This is achieved by dividing the inputs and weights into $G$ groups and normalizing the summation within each group.
At training time, FBN can be viewed as an "inverted bottleneck" mechanism. The intermediate channels are expanded (by a factor of $G$), normalized, and then summarized. This allows for finer-grained coordination during training.
Crucially, at inference time, FBN can be fused into the preceding convolutional layer. This fusion eliminates the need to store intermediate states, effectively making the normalization operation computationally free and without memory overhead. After fusion, Finet networks utilize standard convolutions with an equal channel width, simplifying the architecture and improving efficiency.
The paper also proposes a novel light-weight network architecture called Finet, which leverages FBN. The Finet architecture incorporates depthwise convolutions for parameter reduction and uses FBN within its 1x1 pointwise convolutions. The paper also mentions the optional integration of the Squeeze-Excite module for further performance enhancement.
Datasets
The paper evaluates Finet on the following datasets:
- ImageNet: Used for large-scale image classification.
- CIFAR-10 and CIFAR-100: Used for evaluating performance on smaller image classification tasks and analyzing hyperparameter settings.
Key Insights and Ideas
- Normalizing intermediate states is beneficial: FBN's approach of normalizing intermediate summation states, rather than the final output, leads to improved accuracy, especially when the intermediate channels are wider than the final output channels.
- Inference-time fusion is key to efficiency: The ability to fuse FBN into convolutional layers at inference time is a critical insight. This makes FBN computationally free during inference, a significant advantage for mobile and embedded applications.
- Finet offers a strong trade-off: The Finet architecture, by combining FBN with depthwise convolutions and optional Squeeze-Excite modules, demonstrates a compelling balance between accuracy, FLOPs, and inference speed.
- FBN generalizes to heavy-weight networks: The paper shows that FBN can also improve the performance of larger networks like ResNet, suggesting its broader applicability beyond light-weight architectures.
Key Results
- ImageNet Performance:
- Small Finet (with Squeeze-Excite) achieves 65.706% accuracy with 43M FLOPs.
- Large Finet (with Squeeze-Excite) achieves 73.786% accuracy with 303M FLOPs.
- These results are state-of-the-art, outperforming models like ShuffleNetV2 and MobileNetV2 at similar computational budgets.
- Inference Speed:
- Finet (without Squeeze-Excite) is reported as the fastest model across three different mobile phones compared to ShuffleNetV2, MobileNetV2, MobileNetV3, and MnasNet.
- Finet with Squeeze-Excite is the second fastest.
- CIFAR Performance:
- FBN consistently improves accuracy over BN on CIFAR-10 and CIFAR-100, with the highest accuracy of 93.674% on CIFAR-10 and 77.202% on CIFAR-100 achieved with $G=8$.
- FBN also shows benefits for ResNet18 and ResNet50 on CIFAR datasets.
Limitations and Future Work
- Training Memory Overhead: The paper acknowledges that FBN requires storing intermediate states during training, which can lead to higher memory usage compared to BN.
- Hyperparameter Sensitivity: While the paper explores hyperparameter settings, further investigation into optimal settings for various tasks and architectures might be beneficial.
- Future Work: The authors suggest that FBN can be applied to other network architectures and tasks, and further exploration of its benefits in different domains is warranted. The potential for FBN to improve training stability and generalization in various scenarios is also an area for future research.
Summary updated