ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network

Dongyoon Han, Sangdoo Yun, Byeongho Heo, Young Joon Yoo

Abstract

Designing an efficient model within the limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications.

Introduction: The "Staircase" Bottleneck

In the quest for efficient neural networks, a standard design convention has emerged: maintaining narrow channel dimensions in early stages and expanding them significantly in later stages. When visualized, this configuration resembles a "staircase" or a piecewise linear function. ReXNet challenges this convention, arguing that these sudden jumps and narrow early layers create representational bottlenecks.

Figure 1: Comparison of channel dimension configurations across various lightweight models. The plot illustrates the conventional 'staircase' pattern found in models like MobileNetV2 and EfficientNet-B0.

The authors hypothesize that the expressiveness of a layer—its ability to carry diverse information—is limited by the mathematical rank of its feature matrices. Drastic expansions or overly narrow layers can lead to information loss that cannot be recovered deeper in the network.

Methods: Rank Analysis and Linear Growth

Rank Analysis

The researchers conducted an empirical study of matrix rank across thousands of networks. They discovered that the standard practice of using narrow early layers is mathematically suboptimal. Specifically, if the input-to-output channel ratio is too low, the network cannot produce high-rank (diverse) features.

Figure 5: Average rank ratio across different convolution layers (1x1, 3x3, and Inverted Bottlenecks) and activation functions. The data shows that SiLU (Swish) and higher dimension ratios help maintain feature rank.

Their analysis revealed that complex activation functions like SiLU (Swish) act as "rank expanders," particularly when placed after layers with constrained expansion ratios.

Linear Parameterization

Instead of relying solely on complex Neural Architecture Search (NAS), ReXNet simplifies the search space by modeling channel dimensions ($c_i$) as a linear function of the block index ($i$): $c_i = a \cdot i + b$. This replaces the "staircase" growth with a smooth, linear increase across the entire network. The final ReXNet architecture redesigns the standard inverted bottleneck to incorporate this linear growth, strategic SiLU placement, and Squeeze-and-Excitation (SE) modules.

Key Results

ReXNet achieves state-of-the-art trade-offs between accuracy and computational efficiency:

  • ImageNet Performance: ReXNet (x1.0) achieves 77.9% Top-1 accuracy with only 0.4B FLOPs, outperforming EfficientNet-B0 (77.3%) and MobileNetV3-Large (75.2%) with fewer parameters.
  • Hardware Efficiency: Due to its regular architecture, ReXNet is significantly faster on real hardware than NAS-generated models. ReXNet (x2.0) is approximately 1.4x faster on CPU and 2.0x faster on GPU than EfficientNet-B3 at similar accuracy levels.
  • Downstream Tasks: When used as a backbone for COCO object detection, ReXNet (x0.9) outperformed EfficientNet-B0 by 1.2 points in mAP while maintaining a faster inference speed.

Conclusion

ReXNet demonstrates that the "shape" of a network's width is as critical as its depth or building blocks. By replacing the conventional staircase channel configuration with a simple linear growth rule and optimizing for feature rank, ReXNet provides a powerful new principle for designing the next generation of lightweight, high-performance convolutional neural networks.

Summary updated

Abstract

Designing an efficient model within the limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications. Code and ImageNet pretrained models are available at https://github.com/clovaai/rexnet.
BibTeX
@inproceedings{han2021rexnet,
  author = {Dongyoon Han and Sangdoo Yun and Byeongho Heo and Young Joon Yoo},
  title = {{ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00079},
}
Figure