FDViT: Improve the Hierarchical Architecture of Vision Transformer

Yixing Xu, Chao Li, Dong Li, Xiao Sheng, Fan Jiang, Lu Tian, Ashish SirasaoICCV 2023 · OpenReview

Abstract

Despite the fact that transformer-based models have yielded great success in computer vision tasks, they suffer from the challenge of high computational costs that limits their use on resource-constrained devices. One major reason is that vision transformers have redundant calculations since the self-attention operation generates patches with high similarity at a later stage in the network. Hierarchical architectures have been proposed for vision transformers to alleviate this challenge. However, by shrinking the spatial dimensions to half of the originals with downsampling layers, the challenge is actually overcompensated, as too much information is lost. In this paper, we propose FDViT to improve the hierarchical architecture of the vision transformer by using a flexible downsampling layer that is not limited to integer stride to smoothly reduce the sizes of the middle feature maps. Furthermore, a masked auto-encoder architecture is used to facilitate the training of the proposed flexible downsampling layer and produces informative outputs. Experimental results on benchmark datasets demonstrate that the proposed method can reduce computational costs while increasing classification performance and achieving state-of-the-art results. For example, the proposed FDViT-S model achieves a top-1 accuracy of 81.5%, which is 1.7 percent points higher than the ViT-S model and reduces 39% FLOPs.

Introduction

Vision Transformers (ViTs) have revolutionized computer vision but remain computationally expensive for deployment on resource-constrained devices. A primary source of inefficiency is feature redundancy: as data progresses through the network, patches become increasingly similar, leading to redundant self-attention computations in deeper layers.

Average cosine similarity of patches across layers for ViT and FDViT variants.

To address this, hierarchical architectures typically employ downsampling layers to reduce spatial resolution. However, standard downsampling often uses an integer stride (e.g., stride 2), which aggressively discards 75% of spatial information in a single step. The authors argue that this "all-or-nothing" approach is too aggressive for early network stages where patch similarity is not yet high enough to justify such drastic reduction, leading to significant information loss that hurts final accuracy.

Methods

The paper proposes two primary innovations to create a more efficient and informative hierarchical architecture: Flexible Downsampling and an MAE-based training strategy.

Flexible Downsampling (FD) Layer

Unlike standard convolutional layers that move in integer steps, the FD layer allows for non-integer strides (e.g., $\sqrt{2} \approx 1.4$). This enables a smoother transition in spatial dimensions across more layers, mitigating the data loss associated with aggressive stride-2 downsampling. Since non-integer strides do not align with a discrete pixel grid, the layer utilizes bilinear interpolation to gather information from four auxiliary points surrounding the target coordinate.

Informative Feature Generation via MAE

To ensure that the reduced feature maps retain critical information, FDViT incorporates a Masked Auto-Encoder (MAE) training objective. During training, a binary mask is applied to the input of the FD layer. A lightweight decoder branch is then tasked with reconstructing the original input from these masked, downsampled features.

The FDViT architecture overview, highlighting the Flexible Downsampling (FD) layers and the Masked Auto-Encoder (MAE) training strategy.

This auxiliary task forces the FD layer to act as an efficient encoder that produces highly informative and compact features. Importantly, the decoder and masking process are only used during training and are discarded during inference, ensuring no additional computational overhead for the final model.

Experimental Results

The authors evaluated FDViT across three major computer vision benchmarks: ImageNet-1k, MSCOCO 2017, and ADE20K.

  • Efficiency vs. Accuracy: The FDViT-S model achieved 81.5% top-1 accuracy on ImageNet, which is 1.7% higher than the standard ViT-S while using 39% fewer FLOPs. The smallest model, FDViT-Ti, reached 73.7% accuracy, outperforming DeiT-Ti by 1.5% with less than half the computational power.
  • Downstream Tasks: On MSCOCO, FDViT-S achieved 39.9 mAP, outperforming both the standard ViT-S (36.9) and the hierarchical PiT-S (39.4). On ADE20K, it achieved 44.0 mIoU, a significant jump over the PiT-S baseline (42.6).
  • Ablation Insights: The results demonstrate that gradual spatial reduction is superior; using four FD layers with a stride of 1.4 outperformed using two layers with a stride of 2.0. Additionally, the MAE masking component was found to be essential for forcing the model to learn compressed, high-quality representations.

Conclusion

FDViT demonstrates that gradual, flexible downsampling combined with a reconstruction-based training objective significantly improves the efficiency-accuracy trade-off in Vision Transformers. By avoiding aggressive information loss and forcing the model to learn informative features, FDViT achieves state-of-the-art performance across multiple vision tasks. Future work may explore automated searches for optimal downsampling schedules and more advanced coordinate gathering functions.

Summary updated

Abstract

Despite the fact that transformer-based models have yielded great success in computer vision tasks, they suffer from the challenge of high computational costs that limits their use on resource-constrained devices. One major reason is that vision transformers have redundant calculations since the self-attention operation generates patches with high similarity at a later stage in the network. Hierarchical architectures have been proposed for vision transformers to alleviate this challenge. However, by shrinking the spatial dimensions to half of the originals with downsampling layers, the challenge is actually overcompensated, as too much information is lost. In this paper, we propose FDViT to improve the hierarchical architecture of the vision transformer by using a flexible downsampling layer that is not limited to integer stride to smoothly reduce the sizes of the middle feature maps. Furthermore, a masked auto-encoder architecture is used to facilitate the training of the proposed flexible downsampling layer and produces informative outputs. Experimental results on benchmark datasets demonstrate that the proposed method can reduce computational costs while increasing classification performance and achieving state-of-the-art results. For example, the proposed FDViT-S model achieves a top-1 accuracy of 81.5%, which is 1.7 percent points higher than the ViT-S model and reduces 39% FLOPs.
BibTeX
@inproceedings{xu2023fdvit,
  author = {Yixing Xu and Chao Li and Dong Li and Xiao Sheng and Fan Jiang and Lu Tian and Ashish Sirasao},
  title = {{FDViT: Improve the Hierarchical Architecture of Vision Transformer}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=zQtCfPbmM9},
}
Figure