TResNet: High Performance GPU-Dedicated Architecture

Tal Ridnik, Hussam Lawen, Asaf Noy, Emanuel Ben Baruch, Gilad Sharir, Itamar Friedman

Introduction: The FLOPs Fallacy

In recent years, the computer vision community has largely focused on reducing Floating Point Operations (FLOPs) as the primary metric for model efficiency. This trend led to architectures like EfficientNet and MobileNet, which utilize depthwise convolutions and complex multi-path structures to minimize theoretical computation.

However, TResNet challenges this paradigm by highlighting a critical "blind spot": FLOPs are often a poor proxy for actual GPU throughput. On modern hardware, performance is frequently bottlenecked by memory access costs and network fragmentation rather than raw computation. Consequently, "low-FLOP" models can be slower in practice than traditional, "high-FLOP" architectures like ResNet50. TResNet is designed to bridge this gap, maximizing actual GPU speed through hardware-aware design.

Architectural Refinements

TResNet (GPU-Dedicated ResNet) retains the robust residual structure of ResNet but introduces several key refinements to optimize for GPU hardware assets.

Structural Improvements

  1. SpaceToDepth Stem: Traditional $7 \times 7$ convolution stems are computationally expensive and can lead to information loss. TResNet replaces this with a non-trainable "SpaceToDepth" layer that rearranges spatial blocks into the depth dimension, reducing resolution by 4x without discarding data.
  2. Anti-Alias Downsampling (AA): To improve shift-equivariance and robustness, standard stride-2 convolutions are replaced with a stride-1 convolution followed by a fixed $3 \times 3$ blur filter.
  3. In-Place Activated BatchNorm (Inplace-ABN): By fusing BatchNorm and the Leaky-ReLU activation into a single operation, TResNet significantly reduces the memory footprint of activation maps. This allows for nearly double the batch size compared to standard ResNet50 on the same hardware (e.g., 512 vs 288 on a V100).
  4. Hybrid Block Selection: TResNet employs a non-uniform block strategy. It uses BasicBlocks in early stages to provide a larger receptive field for initial feature extraction, and Bottleneck blocks in later stages to provide the computational capacity needed for complex classification.

Low-Level Optimizations

Beyond structural changes, TResNet leverages code-level enhancements:

  • JIT Compilation: PyTorch's Just-In-Time compiler is used for non-learnable modules to minimize overhead.
  • Fast Global Average Pooling (GAP): A dedicated, optimized implementation of GAP reduces memory access, achieving up to 5x speedup over standard implementations.

Key Results and Performance

TResNet demonstrates that hardware-dedicated design leads to superior accuracy-speed trade-offs across various benchmarks.

Comparison of TResNet-M with standard architectures on ImageNet. TResNet-M achieves higher accuracy than ResNet50 while maintaining superior inference speed.

As shown in the table above, TResNet-M achieves 80.8% Top-1 accuracy on ImageNet, outperforming ResNet50 (79.0%) while maintaining a higher inference throughput of 2930 images/sec.

Accuracy vs. Inference Speed comparison between TResNet and EfficientNet. TResNet models occupy a superior Pareto frontier for GPU deployment.

When compared to the EfficientNet family, TResNet models offer a significantly better Pareto frontier for GPU deployment. For instance, TResNet-XL reaches 84.3% Top-1 accuracy, rivaling state-of-the-art models while being substantially faster to train and deploy. In downstream tasks, TResNet models have shown to be 8x to 15x faster than EfficientNet-B7 while maintaining comparable accuracy. Furthermore, TResNet-L achieved a new state-of-the-art on MS-COCO multi-label classification with 86.4 mAP.

Conclusion

TResNet proves that rethinking architecture through the lens of actual hardware performance rather than theoretical metrics can yield significant gains. By prioritizing memory efficiency, reducing fragmentation, and utilizing dense convolutions that favor GPU Tensor Cores, TResNet provides a powerful alternative for high-performance computer vision applications. Future work may explore how these GPU-centric optimizations translate to other specialized hardware like TPUs or mobile accelerators.

Summary updated

Abstract

Many deep learning models, developed in recent years, reach higher ImageNet accuracy than ResNet50, with fewer or comparable FLOPS count. While FLOPs are often seen as a proxy for network efficiency, when measuring actual GPU training and inference throughput, vanilla ResNet50 is usually significantly faster than its recent competitors, offering better throughput-accuracy trade-off. In this work, we introduce a series of architecture modifications that aim to boost neural networks' accuracy, while retaining their GPU training and inference efficiency. We first demonstrate and discuss the bottlenecks induced by FLOPs-optimizations. We then suggest alternative designs that better utilize GPU structure and assets. Finally, we introduce a new family of GPU-dedicated models, called TResNet, which achieve better accuracy and efficiency than previous ConvNets. Using a TResNet model, with similar GPU throughput to ResNet50, we reach 80.8 top-1 accuracy on ImageNet. Our TResNet models also transfer well and achieve state-of-the-art accuracy on competitive single-label classification datasets such as Stanford cars (96.0%), CIFAR-10 (99.0%), CIFAR-100 (91.5%) and Oxford-Flowers (99.1%). They also perform well on multi-label classification and object detection tasks. Implementation is available at: https://github.com/mrT23/TResNet.
BibTeX
@inproceedings{ridnik2021tresnet,
  author = {Tal Ridnik and Hussam Lawen and Asaf Noy and Emanuel Ben Baruch and Gilad Sharir and Itamar Friedman},
  title = {{TResNet: High Performance GPU-Dedicated Architecture}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=kVgX216cUXM},
}
Figure