Rethinking Channel Dimensions for Efficient Model Design

Dongyoon Han, Sangdoo Yun, Byeongho Heo, Youngjoon Yoo

Abstract

Standard lightweight neural network designs often follow a "stage-wise" convention where channel dimensions increase in discrete steps. Researchers from NAVER AI Lab challenge this paradigm by analyzing the mathematical rank of feature maps. They introduce ReXNet (Rank Expansion Networks), which utilizes a simple linear increase in channel dimensions per block. This approach significantly enhances model expressiveness and accuracy without adding computational overhead, achieving state-of-the-art results on ImageNet and downstream tasks.

Introduction

In mobile-friendly AI, efficiency is paramount. For years, architectures like MobileNetV2 have utilized a "stage-wise" configuration: networks are divided into stages where all layers within a stage share the same channel count. Jumps in channel dimensions (e.g., 16 → 24 → 32) occur only at stage boundaries.

The authors argue this convention is a bottleneck. While designed for FLOP-efficiency, it often leads to rank deficiency. In linear algebra, low-rank output indicates redundant or "collapsed" features, limiting a model's representational capacity. ReXNet explores whether a different channel distribution can maximize expressiveness for a given computational budget.

Methodology: From Rank Analysis to ReXNet

The design of ReXNet is rooted in a first-principles approach to architecture:

Rank Analysis

The researchers empirically studied how layer designs affect the rank of output features. They found that "drastic" channel expansion (such as the 6x expansion in standard inverted bottlenecks) can harm rank, whereas complex nonlinearities like SiLU (Swish) or ELU help preserve it.

Empirical rank analysis across different convolution types and activation functions.

Linear Parameterization and ReXNet Design

Instead of searching for entirely new building blocks, the authors fixed the block type (Inverted Bottleneck) and optimized the channel configuration. Their search revealed a simple, effective pattern: channels should grow as a linear function of the block index rather than in stages.

The ReXNet design follows the rule: $c_i = a \cdot i + b$, where $c_i$ is the channel dimension of the $i$-th block. Additionally, ReLU6 was replaced with SiLU in specific locations to further enhance feature rank.

Comparison of channel configurations and performance across various lightweight architectures.

Key Insights

  • The "Stage-Wise" Myth: Piecewise-linear channel configurations are suboptimal. A smooth, continuous increase in width allows for more effective representational capacity expansion.
  • Rank as a Proxy for Accuracy: There is a strong correlation between the "nuclear norm" (a proxy for rank) of the final feature map and top-1 accuracy. Maintaining high rank generally leads to higher accuracy.
  • Hardware-Friendly Efficiency: While models like EfficientNet are FLOP-efficient, they can be slow on hardware due to complex scaling. ReXNet achieves similar accuracy while being 1.4x to 2.0x faster on CPUs and GPUs because its structure is more hardware-friendly.

Results and Evaluation

ReXNet was validated across several standard benchmarks:

  • ImageNet: ReXNet (x1.0) achieved 77.9% top-1 accuracy, outperforming MobileNetV3 (75.2%) and EfficientNet-B0 (77.3%) with comparable FLOPs.
  • Object Detection (COCO): As a backbone for SSDLite, ReXNet improved AP by +1.2pp over EfficientNet-B0 while using fewer parameters.
  • Scalability: The linear channel rule scales effectively. A larger ReXNet (x2.0) reached 81.6% top-1 accuracy, matching EfficientNet-B3 while running significantly faster.

Conclusion

ReXNet demonstrates that simple modifications to channel configurations can yield significant gains in model performance. By moving away from stage-wise designs toward linear rank expansion, researchers can create more expressive and hardware-efficient models. Future work may involve combining these channel-design principles with advanced Neural Architecture Search (NAS) for building blocks and optimizing for specific hardware latencies.

Summary updated

Abstract

Designing an efficient model within the limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications. Code and ImageNet pretrained models are available at https://github.com/clovaai/rexnet.
BibTeX
@inproceedings{han2021rethinking,
  author = {Dongyoon Han and Sangdoo Yun and Byeongho Heo and Youngjoon Yoo},
  title = {{Rethinking Channel Dimensions for Efficient Model Design}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00079},
}
Figure