HIRI-ViT: Scaling Vision Transformer With High Resolution Inputs

Ting Yao, Yehao Li, Yingwei Pan, Tao Mei

Abstract

Vision Transformers (ViTs) have revolutionized computer vision, yet their application to high-resolution inputs remains constrained by the quadratic computational complexity of self-attention. This paper introduces HIRI-ViT (HIgh-Resolution Inputs ViT), a novel five-stage hybrid architecture designed to maintain efficiency while processing high-resolution imagery. By decomposing early-stage operations into parallel high-resolution and low-resolution CNN branches, HIRI-ViT balances spatial detail with semantic depth. Experimental results demonstrate that HIRI-ViT-S achieves 84.3% Top-1 accuracy on ImageNet-1K with 448×448 inputs at only 5.0 GFLOPs, establishing a new Pareto frontier for high-resolution vision models.

Introduction

In modern computer vision, increasing input resolution is a primary lever for improving performance in tasks requiring fine-grained detail, such as object detection and semantic segmentation. However, the standard Vision Transformer (ViT) architecture faces a significant scaling hurdle: the computational cost of self-attention scales quadratically with the number of input tokens. For instance, scaling a Swin Transformer from 224px to 384px results in a nearly threefold increase in computational cost (from 15.4 to 47.0 GFLOPs).

While hybrid models combining Convolutional Neural Networks (CNNs) and ViTs have attempted to mitigate these costs, they often struggle to maintain efficiency when resolutions scale significantly. HIRI-ViT addresses this by rethinking the standard four-stage architecture and introducing a cost-efficient decomposition strategy for high-resolution feature maps.

Methodology

Five-Stage Hybrid Architecture

Unlike traditional models that utilize a four-stage structure, HIRI-ViT employs a five-stage design. This additional stage allows for a more gradual transition from high-resolution raw pixels to low-resolution semantic tokens, preventing the information loss often associated with aggressive downsampling in early layers.

Two-Branch Decomposition

The core innovation of HIRI-ViT is the two-branch decomposition used in the early stages where resolution is highest. Instead of applying heavy standard operations to the full-resolution map, the model splits the workload:

  1. High-Resolution (HR) Branch: This branch processes the primary features directly but utilizes a minimal number of lightweight convolution operations to minimize GFLOPs.
  2. Low-Resolution (LR) Branch: This branch performs an initial downsampling, followed by more intensive convolutions on the reduced feature map to extract deep semantic information.

The outputs from both branches are subsequently fused, allowing the model to retain fine spatial details from the HR branch while benefiting from the semantic richness of the LR branch.

Efficiency Optimizations

To further enhance inference speed, the authors replaced Layer Normalization (LN) with Batch Normalization (BN) in the first three stages. Unlike LN, BN can be fused into the preceding convolution layers during inference, resulting in a 7.6% improvement in throughput without compromising accuracy. Additionally, the model introduces Inverted Residual Downsampling (IRDS) to facilitate more powerful feature transitions between stages.

EMA Distillation

For training, HIRI-ViT utilizes a specialized EMA Distillation strategy. This approach involves a bi-directional interaction between a "teacher" and a "student" network, leveraging Exponential Moving Average (EMA) to improve performance without the need for massive, externally pre-trained teacher models.

Experimental Results

ImageNet-1K Classification

HIRI-ViT demonstrates superior efficiency across various resolutions. HIRI-ViT-S achieves 84.3% Top-1 accuracy at 448×448 resolution with a budget of only 5.0 GFLOPs. When pushed to 768×768 resolution, the model reaches 85.4% accuracy with 16.1 GFLOPs—a fraction of the compute required by standard ViTs at similar scales. Robustness tests on ImageNet-V2 confirmed that these gains are consistent and not the result of overfitting.

Downstream Tasks

The architecture's focus on high-resolution inputs translates directly to success in dense prediction tasks:

  • Object Detection (COCO): HIRI-ViT-S outperformed the Swin-T baseline by 5.1% AP under similar computational constraints.
  • Semantic Segmentation (ADE20K): The model showed significant improvements in mIoU, validating the importance of the two-branch decomposition for spatial accuracy.

Conclusion and Future Work

HIRI-ViT provides a principled framework for scaling vision models to high-resolution inputs without the typical quadratic explosion in cost. By utilizing a five-stage architecture and a dual-branch CNN/ViT hybrid approach, it achieves state-of-the-art efficiency.

Future research directions include extending these high-resolution principles to Video Vision Transformers, where the addition of a temporal dimension further complicates computational scaling. Additionally, while the model is highly efficient in terms of GFLOPs, further work is needed to optimize the multi-branch structure for even higher raw hardware throughput.

Summary updated

BibTeX
@article{yao2024hiri,
  author = {Ting Yao and Yehao Li and Yingwei Pan and Tao Mei},
  title = {{HIRI-ViT: Scaling Vision Transformer With High Resolution Inputs}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3379457},
}
Figure