Q-HyViT: Post-Training Quantization for Hybrid Vision Transformer with Bridge Block Reconstruction

Jemin Lee, Yongin Kwon, Jeman Park, Misun Yu, Hwanjun Song

Abstract

Hybrid Vision Transformers (ViTs), which combine the local feature extraction of CNNs with the global reasoning of Transformers, offer a promising balance of efficiency and performance. However, deploying these models on resource-constrained IoT devices via post-training quantization (PTQ) often leads to significant accuracy degradation. This paper introduces Q-HyViT, the first PTQ framework specifically designed for hybrid ViTs. By identifying the "bridge block"—the transition between CNN and Transformer layers—as a primary source of error, Q-HyViT employs a novel Bridge Block Reconstruction method. This approach utilizes Hessian-based optimization and automated granularity selection to achieve substantial accuracy gains, including up to a 29.75% improvement for 6-bit quantization on ImageNet-1K.

Introduction

Vision Transformers (ViTs) have set new benchmarks in computer vision but remain computationally intensive for edge deployment. Hybrid architectures like MobileViT and EfficientFormer address this by integrating Convolutional Neural Network (CNN) layers for local processing. Despite their efficiency, these models struggle with standard quantization techniques. Converting 32-bit floating-point (FP32) values to 8-bit or 6-bit integers (W8A8/W6A6) often causes a "drastic accuracy drop" in hybrid models due to highly dynamic activation ranges, zero-point overflows in transitional "bridge blocks," and diverse normalization layers (BatchNorm, LayerNorm, GroupNorm). Furthermore, the small parameter counts of these models (often <5M) leave little room for the precision loss inherent in quantization.

Methods

Q-HyViT addresses these challenges through a unified PTQ framework centered on Bridge Block Reconstruction.

Overview of the Q-HyViT framework illustrating the Bridge Block Reconstruction process.

The framework introduces several key innovations:

  • Hessian-based Optimization: Instead of simple Mean Squared Error (MSE), Q-HyViT uses second-order derivatives (the Hessian) to prioritize minimizing errors that most significantly impact final task performance.
  • Automated Granularity & Scheme Selection: The system dynamically chooses between channel-wise and layer-wise quantization, as well as symmetric or asymmetric schemes. This flexibility is vital for managing the zero-point overflow issues prevalent in bridge blocks.
  • Integrated Reconstruction: Unlike traditional layer-wise PTQ, Q-HyViT optimizes all dependencies within a bridge block simultaneously, ensuring a smooth transition from local CNN representations to global Transformer representations.

Key Results

The framework was evaluated on ImageNet-1K across five hybrid architectures: MobileViTv1/v2, Mobile-Former, and EfficientFormerV1/V2.

Activation distributions for various bridge block configurations, comparing full-precision (FP) and quantized (Quant) models.

Q-HyViT consistently outperformed state-of-the-art PTQ methods:

  • 8-bit Quantization (W8A8): Average accuracy improvement of 17.73%. Notably, for the tiny MobileViTv1-xxs, Q-HyViT maintained 68.20% accuracy, whereas competing methods often failed completely.
  • 6-bit Quantization (W6A6): Average improvement of 29.75%, demonstrating robustness in low-precision regimes where Transformers typically struggle.
  • Fully Quantized Setup: When quantizing non-linear operations like Softmax, Q-HyViT improved accuracy by an average of 43.63% over baseline methods like FQ-ViT.

Conclusion and Future Work

Q-HyViT represents a significant advancement in making hybrid ViTs viable for IoT and "tinyML" applications. By focusing on the critical "bridge" between local and global processing, it recovers substantial accuracy lost during quantization. While Q-HyViT is a major step forward, quantizing "extremely small-sized models" (well under 1M parameters) remains a challenge. Future work will focus on optimizing the framework for specific hardware accelerators and further refining quantization for these ultra-tiny models.

Summary updated

Abstract

Recently, vision transformers (ViTs) have superseded convolutional neural networks in numerous applications, including classification, detection, and segmentation. However, the high computational requirements of ViTs hinder their widespread implementation. To address this issue, researchers have proposed efficient hybrid transformer architectures that combine convolutional and transformer layers with optimized attention computation of linear complexity. Additionally, post-training quantization has been proposed as a means of mitigating computational demands. For mobile devices, achieving optimal acceleration for ViTs necessitates the strategic integration of quantization techniques and efficient hybrid transformer structures. However, no prior investigation has applied quantization to efficient hybrid transformers. In this paper, we discover that applying existing post-training quantization (PTQ) methods for ViTs to efficient hybrid transformers leads to a drastic accuracy drop, attributed to the four following challenges: (i) highly dynamic ranges, (ii) zero-point overflow, (iii) diverse normalization, and (iv) limited model parameters ($<$5M). To overcome these challenges, we propose a new post-training quantization method, which is the first to quantize efficient hybrid ViTs (MobileViTv1, MobileViTv2, Mobile-Former, EfficientFormerV1, EfficientFormerV2). We achieve a significant improvement of 17.73% for 8-bit and 29.75% for 6-bit on average, respectively, compared with existing PTQ methods (EasyQuant, FQ-ViT, PTQ4ViT, and RepQ-ViT)}. We plan to release our code at https://gitlab.com/ones-ai/q-hyvit.
BibTeX
@article{lee2024q,
  author = {Jemin Lee and Yongin Kwon and Jeman Park and Misun Yu and Hwanjun Song},
  title = {{Q-HyViT: Post-Training Quantization for Hybrid Vision Transformer with Bridge Block Reconstruction}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  url = {https://openreview.net/forum?id=rM1kBIvNxh},
}
Figure