Abstract
Designing an efficient model within limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications.
Introduction
In the domain of mobile-first computer vision, lightweight models such as MobileNetV2 and EfficientNet have become the industry standard. These architectures typically adhere to a specific design convention: grouping layers into "stages" where every layer within a stage shares the same number of channels. The channel count then jumps significantly at the start of a new stage, creating a "staircase" or piecewise linear configuration.
Rank Expansion Networks (ReXNet) challenge this long-standing convention. The authors hypothesize that these sudden jumps and periods of stagnation in channel dimensions limit the expressiveness of the network. By replacing traditional staircase-like channel jumps with a simple linear expansion based on block index, ReXNet achieves state-of-the-art performance while maintaining high efficiency on mobile hardware.
The Bottleneck of Stage-wise Configuration
The core argument is that the standard stage-wise channel configuration acts as a bottleneck for model accuracy. This configuration often leads to "rank deficiency," where layers produce redundant information, failing to fully utilize the available FLOP budget.
As shown in the figure above, most existing models (like MobileNetV2 and MNasNet) exhibit a step-like increase in channel dimensions. ReXNet moves away from this by proposing that the channel dimension $c_i$ of the $i$-th block should be a simple linear function of its index: $c_i = a \cdot i + b$.
Rank Analysis and Design Principles
To ground their architectural changes, the researchers used the mathematical matrix rank of output features as a proxy for a layer's expressiveness. Through extensive empirical study, they found that "drastic expansion"—where a layer's output is significantly larger than its input—often leads to rank deficiency.
The rank analysis revealed several key insights:
- Non-linearities Matter: More complex non-linearities like SiLU (Swish) or ELU are more effective at expanding the rank of features in 1x1 and 3x3 convolutions compared to standard ReLU.
- Rank as a Guide: Models with higher "nuclear norms" (a proxy for rank) consistently achieve higher accuracy.
- Linear Parameterization: Instead of relying solely on expensive Neural Architecture Search (NAS), a simple linear expansion rule can effectively distribute channels to maximize information flow.
Experimental Results
ReXNet was validated across several standard benchmarks, demonstrating superior performance and efficiency.
ImageNet-1K Classification
ReXNet (x1.0) achieved a 77.9% top-1 accuracy, surpassing EfficientNet-B0 (77.3%) with a similar FLOP count. When scaled, ReXNet (x2.0) proved to be significantly faster than comparable models, running approximately 1.4x faster on CPU and 2.0x faster on GPU than EfficientNet-B3.
Downstream Tasks
The benefits of ReXNet extend to object detection and segmentation. On the COCO dataset, ReXNet-based SSDLite improved box AP by +1.2 to +1.8 points over EfficientNet-B0 and B1 counterparts. Furthermore, ReXNet consistently outperformed ResNet-50 on fine-grained classification tasks like Stanford Cars and Oxford Flowers-102, despite having significantly fewer parameters.
Conclusion
ReXNet demonstrates that a fundamental rethink of channel dimension configuration can yield significant gains in efficiency and accuracy. By moving from a heuristic "staircase" design to a principled linear expansion guided by rank analysis, the researchers have provided a new template for lightweight model design. While the linear rule is a heuristic, its simplicity and effectiveness suggest that continuous channel growth is a powerful alternative to traditional stage-wise architectures.
Summary updated