Abstract
Compact neural networks designed for mobile devices often struggle with limited parameter counts, leading to insufficient feature extraction and "random predictions" during training. We propose Bias Loss, a novel objective function that prioritizes samples with high feature diversity, and SkipblockNet, an architecture utilizing specialized skip connections to preserve low-level descriptors. Our approach achieves state-of-the-art performance on ImageNet, outperforming MobileNetV3 Large by 1% Top-1 accuracy with similar computational costs.
Introduction
Large-scale Convolutional Neural Networks (CNNs) leverage millions of parameters to capture a rich variety of features. In contrast, compact models like MobileNet or ShuffleNet face a "feature thinning" problem where limited parameters fail to extract enough unique descriptors for complex images. This often results in "random predictions" during training, where the model guesses due to a lack of informative features. Standard cross-entropy loss treats these noisy signals with the same weight as high-quality predictions, potentially hindering the optimization process for mobile-efficient architectures.
Proposed Methods
To address these challenges, we introduce a dual-pronged approach involving a new loss function and a refined architecture.
Bias Loss
Bias Loss is designed to focus the optimizer's attention on "valuable" data points characterized by high feature diversity. We utilize the variance of feature maps in the final convolutional layer as a proxy for this diversity. High variance suggests the presence of unique descriptors, while low variance indicates the model is struggling. By down-weighting low-variance samples and up-weighting high-variance ones, Bias Loss prevents "random" guesses from polluting the learning signal.
SkipblockNet Architecture
To mitigate feature loss in deep, narrow networks, we propose SkipblockNet. This architecture incorporates "Skip Blocks" that facilitate the direct transfer of low-level features from initial layers to the final stages of the network.
As shown in the architecture diagram, these skip connections act as a high-speed delivery mechanism, ensuring that the final classification layers have access to a richer set of descriptors, thereby reducing the likelihood of feature-starved random predictions.
Experimental Results
We evaluated our methods across several benchmarks, including ImageNet (ILSVRC-2012), CIFAR-100, and Food-101.
ImageNet Performance
SkipblockNet demonstrates superior efficiency compared to existing state-of-the-art mobile models. Specifically, SkipblockNet-M achieves 76.2% Top-1 accuracy with 246M FLOPs, surpassing MobileNetV3 Large by 1% at a comparable computational budget.
Generalization and Transfer Learning
The benefits of Bias Loss extend beyond our specific architecture. When applied to ShuffleNetV2 0.5× on CIFAR-100, it yielded a 1.5% accuracy improvement over standard cross-entropy without increasing inference-time parameters. Furthermore, SkipblockNet showed strong transfer learning capabilities on the Food-101 dataset and improved object detection performance when used as a backbone for SSD on PASCAL VOC.
Conclusion
Bias Loss and SkipblockNet provide a robust framework for enhancing the performance of compact neural networks. By prioritizing informative samples and preserving feature richness through skip connections, we achieve significant accuracy gains in resource-constrained environments. Future work will explore the use of AutoML to further optimize hyperparameter selection and architecture design.
Summary updated