TL;DR
This paper tackles the inference latency bottleneck in Swin Transformers, a popular vision model. By replacing the computationally expensive GELU activation function with the simpler ReLU and employing knowledge distillation, they achieve significant speedups (over 11%) on NVIDIA GPUs while maintaining high accuracy.
What Problem Does This Paper Address?
Vision Transformers (ViTs) have revolutionized computer vision, offering state-of-the-art performance. However, their computational demands, particularly for inference, can be a significant hurdle, especially for real-time applications or deployment on resource-constrained devices. Swin Transformers, a prominent ViT variant, improve efficiency by using shifted windows for attention, but still suffer from slower inference compared to other deep neural networks. Integer quantization is a common technique to speed up inference by using lower-bit representations for model weights and activations. While previous work has successfully quantized linear components of Swin Transformers, non-linear components like the GELU activation function remain challenging to quantize efficiently without sacrificing accuracy. These non-linearities often require floating-point operations, introducing overhead from quantization/de-quantization steps. This paper aims to address this by finding a way to efficiently quantize the GELU activation in Swin Transformers to reduce inference latency.
Methods
The core of this paper's contribution lies in its approach to handling the GELU activation function within an integer-quantized Swin Transformer. Instead of relying on complex approximations or leaving GELU as a floating-point operation, the authors propose replacing it with the Rectified Linear Unit (ReLU) activation function. ReLU is chosen for its simplicity, low computational complexity, and ease of quantization.
To compensate for the potential accuracy drop introduced by this replacement, the authors employ iterative knowledge distillation. This process involves training the modified "GELU-less" Swin Transformer to mimic the output of the original, full-precision Swin Transformer. The distillation is applied gradually, block by block, to minimize model divergence.
Finally, the resulting GELU-less Swin Transformer is subjected to post-training quantization using the FasterTransformer framework, which quantizes weights and activations to 8-bit integers. The authors also note that they remove the bias fused with the GELU activation to fully eliminate the floating-point operation.
Datasets
The paper uses the ImageNet dataset for evaluating the accuracy of their proposed method. They mention using a portion of the ImageNet training dataset for knowledge distillation.
Key Insights and Ideas
- ReLU as a GELU Replacement: The most significant insight is the successful replacement of the GELU activation with ReLU in Swin Transformers for integer quantization. This highlights that simpler, more quantizable non-linearities can be effective substitutes, even in complex architectures.
- Knowledge Distillation is Crucial: The authors demonstrate that knowledge distillation is essential for maintaining accuracy after replacing GELU with ReLU. Without it, the accuracy drop is significant.
- Focus on Fused Operations: The paper emphasizes that for significant latency improvements in quantized models, it's often necessary to modify or remove entire fused operations rather than just individual components. This is because the overhead of quantization/de-quantization can dominate.
- Bias Removal for Full Integerization: Removing the bias fused with the GELU activation is a necessary step to achieve a fully integerized operation and maximize latency reduction.
Key Results
The proposed GELU-less Swin Transformer, when quantized to 8-bit integers, achieves:
- Inference Latency Reduction: At least an 11% improvement in inference latency compared to the FasterTransformer's quantized Swin model. For specific models like SWINTINY, the speedup is as high as 4.02x compared to the baseline FP32.
- Accuracy Preservation: An accuracy drop of under 0.5% on the ImageNet evaluation dataset. For SWINBASE, their method even achieved a slight accuracy increase (83.4% to 84.6%).
- Hardware: Experiments were conducted on an NVIDIA RTX 4090 GPU.
Limitations and Future Work
- Hardware Dependency: The speedups are reported on a specific NVIDIA GPU (RTX 4090). The actual performance gains might vary on different hardware architectures, especially those with less optimized integer arithmetic support.
- Knowledge Distillation Overhead: While effective, knowledge distillation adds computational cost and time during the training phase.
- Future Work: The paper doesn't explicitly detail future work, but the success of this approach suggests further exploration into quantizing other non-linearities or applying this method to different transformer architectures could be promising. Investigating the impact on other hardware platforms would also be valuable.
Summary updated