Rethinking Channel Dimensions for Efficient Model Design

Dongyoon Han, Sangdoo Yun, Byeongho Heo, Youngjoon Yoo

Abstract

Designing an efficient model within limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications.

Introduction

In the domain of mobile-first computer vision, lightweight models such as MobileNetV2 and EfficientNet have become the industry standard. These architectures typically adhere to a specific design convention: grouping layers into "stages" where every layer within a stage shares the same number of channels. The channel count then jumps significantly at the start of a new stage, creating a "staircase" or piecewise linear configuration.

Rank Expansion Networks (ReXNet) challenge this long-standing convention. The authors hypothesize that these sudden jumps and periods of stagnation in channel dimensions limit the expressiveness of the network. By replacing traditional staircase-like channel jumps with a simple linear expansion based on block index, ReXNet achieves state-of-the-art performance while maintaining high efficiency on mobile hardware.

The Bottleneck of Stage-wise Configuration

The core argument is that the standard stage-wise channel configuration acts as a bottleneck for model accuracy. This configuration often leads to "rank deficiency," where layers produce redundant information, failing to fully utilize the available FLOP budget.

Channel dimension configurations of various state-of-the-art lightweight models. Note the characteristic 'staircase' pattern that ReXNet aims to replace with a continuous linear expansion.

As shown in the figure above, most existing models (like MobileNetV2 and MNasNet) exhibit a step-like increase in channel dimensions. ReXNet moves away from this by proposing that the channel dimension $c_i$ of the $i$-th block should be a simple linear function of its index: $c_i = a \cdot i + b$.

Rank Analysis and Design Principles

To ground their architectural changes, the researchers used the mathematical matrix rank of output features as a proxy for a layer's expressiveness. Through extensive empirical study, they found that "drastic expansion"—where a layer's output is significantly larger than its input—often leads to rank deficiency.

Average rank ratio versus dimension ratio for different convolution types and activation functions, showing that certain non-linearities better preserve feature rank.

The rank analysis revealed several key insights:

  1. Non-linearities Matter: More complex non-linearities like SiLU (Swish) or ELU are more effective at expanding the rank of features in 1x1 and 3x3 convolutions compared to standard ReLU.
  2. Rank as a Guide: Models with higher "nuclear norms" (a proxy for rank) consistently achieve higher accuracy.
  3. Linear Parameterization: Instead of relying solely on expensive Neural Architecture Search (NAS), a simple linear expansion rule can effectively distribute channels to maximize information flow.

Experimental Results

ReXNet was validated across several standard benchmarks, demonstrating superior performance and efficiency.

ImageNet-1K Classification

ReXNet (x1.0) achieved a 77.9% top-1 accuracy, surpassing EfficientNet-B0 (77.3%) with a similar FLOP count. When scaled, ReXNet (x2.0) proved to be significantly faster than comparable models, running approximately 1.4x faster on CPU and 2.0x faster on GPU than EfficientNet-B3.

Downstream Tasks

The benefits of ReXNet extend to object detection and segmentation. On the COCO dataset, ReXNet-based SSDLite improved box AP by +1.2 to +1.8 points over EfficientNet-B0 and B1 counterparts. Furthermore, ReXNet consistently outperformed ResNet-50 on fine-grained classification tasks like Stanford Cars and Oxford Flowers-102, despite having significantly fewer parameters.

Conclusion

ReXNet demonstrates that a fundamental rethink of channel dimension configuration can yield significant gains in efficiency and accuracy. By moving from a heuristic "staircase" design to a principled linear expansion guided by rank analysis, the researchers have provided a new template for lightweight model design. While the linear rule is a heuristic, its simplicity and effectiveness suggest that continuous channel growth is a powerful alternative to traditional stage-wise architectures.

Summary updated

Abstract

Designing an efficient model within the limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications. Code and ImageNet pretrained models are available at https://github.com/clovaai/rexnet.
BibTeX
@inproceedings{han2021rethinking,
  author = {Dongyoon Han and Sangdoo Yun and Byeongho Heo and Youngjoon Yoo},
  title = {{Rethinking Channel Dimensions for Efficient Model Design}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00079},
}
Figure