Bias Loss for Mobile Neural Networks

Lusine Abrahamyan, Valentin Ziatchin, Yiming Chen, Nikos Deligiannis

Abstract

Compact neural networks designed for mobile devices often struggle with limited parameter counts, leading to insufficient feature extraction and "random predictions" during training. We propose Bias Loss, a novel objective function that prioritizes samples with high feature diversity, and SkipblockNet, an architecture utilizing specialized skip connections to preserve low-level descriptors. Our approach achieves state-of-the-art performance on ImageNet, outperforming MobileNetV3 Large by 1% Top-1 accuracy with similar computational costs.

Introduction

Large-scale Convolutional Neural Networks (CNNs) leverage millions of parameters to capture a rich variety of features. In contrast, compact models like MobileNet or ShuffleNet face a "feature thinning" problem where limited parameters fail to extract enough unique descriptors for complex images. This often results in "random predictions" during training, where the model guesses due to a lack of informative features. Standard cross-entropy loss treats these noisy signals with the same weight as high-quality predictions, potentially hindering the optimization process for mobile-efficient architectures.

Proposed Methods

To address these challenges, we introduce a dual-pronged approach involving a new loss function and a refined architecture.

Bias Loss

Bias Loss is designed to focus the optimizer's attention on "valuable" data points characterized by high feature diversity. We utilize the variance of feature maps in the final convolutional layer as a proxy for this diversity. High variance suggests the presence of unique descriptors, while low variance indicates the model is struggling. By down-weighting low-variance samples and up-weighting high-variance ones, Bias Loss prevents "random" guesses from polluting the learning signal.

SkipblockNet Architecture

To mitigate feature loss in deep, narrow networks, we propose SkipblockNet. This architecture incorporates "Skip Blocks" that facilitate the direct transfer of low-level features from initial layers to the final stages of the network.

Architecture of SkipblockNet featuring Skip Blocks and Inverted-Residual blocks.

As shown in the architecture diagram, these skip connections act as a high-speed delivery mechanism, ensuring that the final classification layers have access to a richer set of descriptors, thereby reducing the likelihood of feature-starved random predictions.

Experimental Results

We evaluated our methods across several benchmarks, including ImageNet (ILSVRC-2012), CIFAR-100, and Food-101.

ImageNet Performance

SkipblockNet demonstrates superior efficiency compared to existing state-of-the-art mobile models. Specifically, SkipblockNet-M achieves 76.2% Top-1 accuracy with 246M FLOPs, surpassing MobileNetV3 Large by 1% at a comparable computational budget.

Comparison of SkipblockNet (labeled as SkipNet) against other mobile architectures on ImageNet.

Generalization and Transfer Learning

The benefits of Bias Loss extend beyond our specific architecture. When applied to ShuffleNetV2 0.5× on CIFAR-100, it yielded a 1.5% accuracy improvement over standard cross-entropy without increasing inference-time parameters. Furthermore, SkipblockNet showed strong transfer learning capabilities on the Food-101 dataset and improved object detection performance when used as a backbone for SSD on PASCAL VOC.

Conclusion

Bias Loss and SkipblockNet provide a robust framework for enhancing the performance of compact neural networks. By prioritizing informative samples and preserving feature richness through skip connections, we achieve significant accuracy gains in resource-constrained environments. Future work will explore the use of AutoML to further optimize hyperparameter selection and architecture design.

Summary updated

Abstract

Compact convolutional neural networks (CNNs) have witnessed exceptional improvements in performance in recent years. However, they still fail to provide the same predictive power as CNNs with a large number of parameters. The diverse and even abundant features captured by the layers is an important characteristic of these successful CNNs. However, differences in this characteristic between large CNNs and their compact counterparts have rarely been investigated. In compact CNNs, due to the limited number of parameters, abundant features are unlikely to be obtained, and feature diversity becomes an essential characteristic. Diverse features present in the activation maps derived from a data point during model inference may indicate the presence of a set of unique descriptors necessary to distinguish between objects of different classes. In contrast, data points with low feature diversity may not provide a sufficient amount of unique descriptors to make a valid prediction; we refer to them as random predictions. Random predictions can negatively impact the optimization process and harm the final performance. This paper proposes addressing the problem raised by random predictions by reshaping the standard cross-entropy to make it biased toward data points with a limited number of unique descriptive features. Our novel Bias Loss focuses the training on a set of valuable data points and prevents the vast number of samples with poor learning features from misleading the optimization process. Furthermore, to show the importance of diversity, we present a family of SkipNet models whose architectures are brought to boost the number of unique descriptors in the last layers. Our Skipnet-M can achieve 1% higher classification accuracy than MobileNetV3 Large.
BibTeX
@inproceedings{abrahamyan2021bias,
  author = {Lusine Abrahamyan and Valentin Ziatchin and Yiming Chen and Nikos Deligiannis},
  title = {{Bias Loss for Mobile Neural Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00649},
}
Figure