Q-HyViT: Post-Training Quantization of Hybrid Vision Transformers With Bridge Block Reconstruction for IoT Systems

Jemin Lee, Yongin Kwon, Sihyeong Park, Misun Yu, Jeman Park, Hwanjun Song

TL;DR

This paper introduces Q-HyViT, a novel post-training quantization (PTQ) method specifically designed for hybrid vision transformers (HyViTs) used in resource-constrained IoT devices. It addresses four key challenges that cause accuracy degradation when applying existing PTQ methods to HyViTs, achieving significant accuracy improvements over prior art.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have shown great promise in computer vision tasks, but their high computational demands limit their deployment on edge devices like those in the Internet of Things (IoT). To tackle this, researchers have developed "hybrid" ViTs (HyViTs) that combine convolutional and transformer layers for better efficiency. However, quantizing these HyViTs (reducing the precision of model weights and activations to save memory and computation) is tricky. Existing PTQ methods, which are applied after a model is trained, perform poorly on HyViTs, leading to a significant drop in accuracy. This paper identifies four specific reasons for this failure: highly dynamic activation ranges, zero-point overflow issues in "bridge blocks" (components connecting convolutional and transformer parts), diverse normalization techniques used in HyViTs, and the fact that many efficient HyViTs have fewer than 5 million parameters.

Methods

The core of this paper is the proposed Q-HyViT method, a specialized PTQ framework for HyViTs. It tackles the identified challenges through a hybrid reconstruction error minimization technique. Instead of solely focusing on the task loss, Q-HyViT aims to minimize the reconstruction error of the model's output. The key innovations include:

  • Layer-specific reconstruction: The reconstruction strategy differs for "bridge blocks" compared to standard convolutional or transformer layers.
  • Automated optimization: Q-HyViT automatically determines the optimal:
  • Scaling factors: Crucial for mapping full-precision values to quantized ones.
  • Granularity: Whether to apply quantization per-channel or per-layer, addressing dynamic activation ranges.
  • Quantization scheme: Symmetric or asymmetric, to handle zero-point overflow.
  • Hessian-based approach: Leverages second-order information (Hessian matrix approximation) to guide the optimization process, aiming for better accuracy preservation.

The method is implemented within a three-stage algorithm: calibration (forward/backward passes to gather information), and optimization (iteratively refining scaling factors, granularity, and scheme).

Datasets

The paper uses the ImageNet-1K dataset for evaluating the performance of the quantized models. No new datasets or benchmarks are introduced.

Key Insights and Ideas

  • HyViTs are fundamentally different for quantization: Existing PTQ methods designed for pure ViTs or CNNs fail on HyViTs due to their unique architectural components like bridge blocks and diverse normalization layers.
  • Bridge blocks are critical bottlenecks: These connecting layers are particularly susceptible to quantization errors, especially regarding zero-point overflow, due to the transition between local and global representations.
  • Granularity is key, but not always channel-wise: While channel-wise quantization often helps with dynamic ranges, it can exacerbate zero-point overflow in bridge blocks. Q-HyViT's ability to dynamically choose between channel-wise and layer-wise granularity is crucial.
  • Small models are more sensitive: Models with fewer than 5 million parameters are significantly more impacted by quantization, making specialized methods like Q-HyViT essential for their efficient deployment.
  • Full quantization is achievable: The paper demonstrates that even non-linear operations like softmax and normalization can be quantized without catastrophic accuracy loss when using a tailored approach like Q-HyViT.

Key Results

Q-HyViT demonstrates significant improvements over existing PTQ methods (EasyQuant, FQ-ViT, PTQ4ViT, and RepQ-ViT) on five popular hybrid ViT architectures (MobileViTv1, MobileViTv2, Mobile-Former, EfficientFormerV1, and EfficientFormerV2).

  • 8-bit quantization: Achieves an average improvement of 9.54% over EasyQuant and 7.09% over PTQ4ViT.
  • 6-bit quantization: Achieves an average improvement of 43.39% over EasyQuant and 8.65% over PTQ4ViT.
  • Fully quantized models (including softmax/normalization): Achieves an average accuracy improvement of 43.63% compared to FQ-ViT.
  • Sub-5M parameter models: Q-HyViT shows less than 1% accuracy drop for 8-bit quantization on these challenging models.

Limitations and Future Work

  • Running Time: Q-HyViT has a longer running time compared to some non-reconstruction-based methods (like RepQ-ViT), although this is considered a minor trade-off for offline quantization.
  • Extremely Small Models: While Q-HyViT significantly improves quantization for sub-5M models, extremely small-sized models still present a difficult challenge, suggesting the need for further exploration of novel approaches.
  • Hardware Integration: The paper briefly touches upon the implications for IoT hardware design but doesn't delve into specific hardware co-design aspects. Future work could explore tighter integration with hardware accelerators.

Summary updated

Abstract

Recently, vision transformers (ViTs) have superseded convolutional neural networks in numerous applications, including classification, detection, and segmentation. However, the high computational requirements of ViTs hinder their widespread implementation. To address this issue, researchers have proposed efficient hybrid transformer architectures that combine convolutional and transformer layers with optimized attention computation of linear complexity. Additionally, posttraining quantization (PTQ) has been proposed as a means of mitigating computational demands. For mobile devices, achieving optimal acceleration for ViTs necessitates the strategic integration of quantization techniques and efficient hybrid transformer structures. However, no prior investigation has applied quantization to efficient hybrid transformers. In this article, we discover that applying existing PTQ methods for ViTs to efficient hybrid transformers leads to a drastic accuracy drop, attributed to the four following challenges: 1) highly dynamic ranges; 2) zero-point overflow; 3) diverse normalization; and 4) limited model parameters (<5M). To overcome these challenges, we propose a new PTQ method, which is the first to quantize efficient hybrid ViTs (MobileViTv1, MobileViTv2, Mobile-Former, EfficientFormerV1, and EfficientFormerV2). We achieve a significant improvement of 17.73% for 8-bit and 29.75% for 6-bit on average, respectively, compared with the existing PTQ methods (EasyQuant, FQ-ViT, PTQ4ViT, and RepQ-ViT). We plan to release our code at https://gitlab.com/ones-ai/q-hyvit.
BibTeX
@article{lee2024q,
  author = {Jemin Lee and Yongin Kwon and Sihyeong Park and Misun Yu and Jeman Park and Hwanjun Song},
  title = {{Q-HyViT: Post-Training Quantization of Hybrid Vision Transformers With Bridge Block Reconstruction for IoT Systems}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  url = {https://openreview.net/forum?id=rM1kBIvNxh},
}
Figure