Abstract
Standard lightweight neural network designs often follow a "stage-wise" convention where channel dimensions increase in discrete steps. Researchers from NAVER AI Lab challenge this paradigm by analyzing the mathematical rank of feature maps. They introduce ReXNet (Rank Expansion Networks), which utilizes a simple linear increase in channel dimensions per block. This approach significantly enhances model expressiveness and accuracy without adding computational overhead, achieving state-of-the-art results on ImageNet and downstream tasks.
Introduction
In mobile-friendly AI, efficiency is paramount. For years, architectures like MobileNetV2 have utilized a "stage-wise" configuration: networks are divided into stages where all layers within a stage share the same channel count. Jumps in channel dimensions (e.g., 16 → 24 → 32) occur only at stage boundaries.
The authors argue this convention is a bottleneck. While designed for FLOP-efficiency, it often leads to rank deficiency. In linear algebra, low-rank output indicates redundant or "collapsed" features, limiting a model's representational capacity. ReXNet explores whether a different channel distribution can maximize expressiveness for a given computational budget.
Methodology: From Rank Analysis to ReXNet
The design of ReXNet is rooted in a first-principles approach to architecture:
Rank Analysis
The researchers empirically studied how layer designs affect the rank of output features. They found that "drastic" channel expansion (such as the 6x expansion in standard inverted bottlenecks) can harm rank, whereas complex nonlinearities like SiLU (Swish) or ELU help preserve it.
Linear Parameterization and ReXNet Design
Instead of searching for entirely new building blocks, the authors fixed the block type (Inverted Bottleneck) and optimized the channel configuration. Their search revealed a simple, effective pattern: channels should grow as a linear function of the block index rather than in stages.
The ReXNet design follows the rule: $c_i = a \cdot i + b$, where $c_i$ is the channel dimension of the $i$-th block. Additionally, ReLU6 was replaced with SiLU in specific locations to further enhance feature rank.
Key Insights
- The "Stage-Wise" Myth: Piecewise-linear channel configurations are suboptimal. A smooth, continuous increase in width allows for more effective representational capacity expansion.
- Rank as a Proxy for Accuracy: There is a strong correlation between the "nuclear norm" (a proxy for rank) of the final feature map and top-1 accuracy. Maintaining high rank generally leads to higher accuracy.
- Hardware-Friendly Efficiency: While models like EfficientNet are FLOP-efficient, they can be slow on hardware due to complex scaling. ReXNet achieves similar accuracy while being 1.4x to 2.0x faster on CPUs and GPUs because its structure is more hardware-friendly.
Results and Evaluation
ReXNet was validated across several standard benchmarks:
- ImageNet: ReXNet (x1.0) achieved 77.9% top-1 accuracy, outperforming MobileNetV3 (75.2%) and EfficientNet-B0 (77.3%) with comparable FLOPs.
- Object Detection (COCO): As a backbone for SSDLite, ReXNet improved AP by +1.2pp over EfficientNet-B0 while using fewer parameters.
- Scalability: The linear channel rule scales effectively. A larger ReXNet (x2.0) reached 81.6% top-1 accuracy, matching EfficientNet-B3 while running significantly faster.
Conclusion
ReXNet demonstrates that simple modifications to channel configurations can yield significant gains in model performance. By moving away from stage-wise designs toward linear rank expansion, researchers can create more expressive and hardware-efficient models. Future work may involve combining these channel-design principles with advanced Neural Architecture Search (NAS) for building blocks and optimizing for specific hardware latencies.
Summary updated