Abstract
Hybrid Vision Transformers (ViTs), which combine the local feature extraction of CNNs with the global reasoning of Transformers, offer a promising balance of efficiency and performance. However, deploying these models on resource-constrained IoT devices via post-training quantization (PTQ) often leads to significant accuracy degradation. This paper introduces Q-HyViT, the first PTQ framework specifically designed for hybrid ViTs. By identifying the "bridge block"—the transition between CNN and Transformer layers—as a primary source of error, Q-HyViT employs a novel Bridge Block Reconstruction method. This approach utilizes Hessian-based optimization and automated granularity selection to achieve substantial accuracy gains, including up to a 29.75% improvement for 6-bit quantization on ImageNet-1K.
Introduction
Vision Transformers (ViTs) have set new benchmarks in computer vision but remain computationally intensive for edge deployment. Hybrid architectures like MobileViT and EfficientFormer address this by integrating Convolutional Neural Network (CNN) layers for local processing. Despite their efficiency, these models struggle with standard quantization techniques. Converting 32-bit floating-point (FP32) values to 8-bit or 6-bit integers (W8A8/W6A6) often causes a "drastic accuracy drop" in hybrid models due to highly dynamic activation ranges, zero-point overflows in transitional "bridge blocks," and diverse normalization layers (BatchNorm, LayerNorm, GroupNorm). Furthermore, the small parameter counts of these models (often <5M) leave little room for the precision loss inherent in quantization.
Methods
Q-HyViT addresses these challenges through a unified PTQ framework centered on Bridge Block Reconstruction.
The framework introduces several key innovations:
- Hessian-based Optimization: Instead of simple Mean Squared Error (MSE), Q-HyViT uses second-order derivatives (the Hessian) to prioritize minimizing errors that most significantly impact final task performance.
- Automated Granularity & Scheme Selection: The system dynamically chooses between channel-wise and layer-wise quantization, as well as symmetric or asymmetric schemes. This flexibility is vital for managing the zero-point overflow issues prevalent in bridge blocks.
- Integrated Reconstruction: Unlike traditional layer-wise PTQ, Q-HyViT optimizes all dependencies within a bridge block simultaneously, ensuring a smooth transition from local CNN representations to global Transformer representations.
Key Results
The framework was evaluated on ImageNet-1K across five hybrid architectures: MobileViTv1/v2, Mobile-Former, and EfficientFormerV1/V2.
Q-HyViT consistently outperformed state-of-the-art PTQ methods:
- 8-bit Quantization (W8A8): Average accuracy improvement of 17.73%. Notably, for the tiny MobileViTv1-xxs, Q-HyViT maintained 68.20% accuracy, whereas competing methods often failed completely.
- 6-bit Quantization (W6A6): Average improvement of 29.75%, demonstrating robustness in low-precision regimes where Transformers typically struggle.
- Fully Quantized Setup: When quantizing non-linear operations like Softmax, Q-HyViT improved accuracy by an average of 43.63% over baseline methods like FQ-ViT.
Conclusion and Future Work
Q-HyViT represents a significant advancement in making hybrid ViTs viable for IoT and "tinyML" applications. By focusing on the critical "bridge" between local and global processing, it recovers substantial accuracy lost during quantization. While Q-HyViT is a major step forward, quantizing "extremely small-sized models" (well under 1M parameters) remains a challenge. Future work will focus on optimizing the framework for specific hardware accelerators and further refining quantization for these ultra-tiny models.
Summary updated