Analysis of Quantization on MLP-based Vision Models

Lingran Zhao, Zhen Dong, Kurt KeutzerarXiv 2022 · DOI · arXiv

Introduction

Multi-Layer Perceptron (MLP) based vision models, such as MLP-Mixer and ResMLP, have emerged as efficient alternatives to Convolutional Neural Networks (CNNs) and Transformers. Their structural simplicity and high parallelism offer significant throughput advantages on modern hardware. However, deploying these models on edge devices typically requires quantization—converting 32-bit floating-point weights and activations into low-bit integers (e.g., 8-bit or 4-bit). Unlike CNNs, MLP-based models often experience catastrophic accuracy degradation when quantized. This paper investigates the root causes of this sensitivity and proposes a "quantization-friendly" toolkit to enable robust low-precision deployment.

The Quantization Challenge: Activation Outliers and Layer Sensitivity

The authors identify two primary obstacles to effective MLP quantization: extreme activation ranges and the inherent sensitivity of token-mixing layers.

1. Exploding Activation Ranges

Standard MLP architectures can produce activation values with extreme outliers. These outliers "stretch" the quantization grid, forcing the majority of critical data into a very narrow range of integer values, which leads to significant information loss. As shown in Figure 5, models like ResMLP and ConvMixer exhibit much higher maximum and top-1% activation values compared to ResNet or ViT.

Activation value analysis across different model architectures and the impact of proposed normalization/clipping techniques.

2. Sensitive Token-Mixing Layers

Through Hessian trace analysis, the researchers found that token-mixing MLPs—which process spatial information across patches—are significantly more sensitive to quantization noise than channel-mixing layers. This sensitivity stems from the heavy reuse of parameters across all channels, where a small error in a shared weight can propagate across the entire feature map.

Proposed Solutions: The Q-MLP Toolkit

To address these issues, the authors introduce several architectural and algorithmic improvements:

  • Restricting Activation Ranges: To stabilize activations, simple Affine layers are replaced with LayerNorm or BatchNorm. Additionally, unbounded activation functions (ReLU/GELU) are replaced with Parametrized Clipping Activation (PACT), which learns an optimal upper bound during training to clip outliers.
  • Multi-Token-Mixing: To reduce the sensitivity of spatial processing, the authors propose dividing channels into groups and applying independent token-mixing MLPs to each group. This "multi-token-mixing" approach (illustrated in Figure 1) increases model expressivity and robustness without a substantial increase in parameter count.
Comparison between standard single-token-mixing and the proposed multi-token-mixing architecture.
  • Advanced Quantization Schemes: The toolkit includes percentile quantization (clipping the top 1% of values) and asymmetric linear quantizers to better accommodate skewed activation distributions.

Key Results and Performance

The resulting Q-MLP models demonstrate that MLP-based architectures can indeed be quantized effectively:

  • 8-bit Quantization: Achieved 79.68% accuracy on ImageNet-1k with a model size of only 30 MB.
  • 4-bit Quantization: Maintained a competitive 78.47% accuracy (15 MB model size), significantly outperforming standard post-training quantization (PTQ) baselines.
  • Hardware Efficiency: On a TITAN RTX GPU, the ResMLP-S24/16 variant achieved a throughput of 468 images/second, surpassing both ViT-S/16 (451 img/sec) and ResNet-50 (466 img/sec), validating the efficiency benefits of the MLP architecture.

Conclusion

By addressing the unique architectural sensitivities of MLP-based models, this work paves the way for their deployment in resource-constrained environments. The combination of activation range restriction and multi-token-mixing allows these models to retain their throughput advantages while achieving the memory efficiency of 4-bit and 8-bit quantization.

Summary updated

Abstract

Quantization is wildly taken as a model compression technique, which obtains efficient models by converting floating-point weights and activations in the neural network into lower-bit integers. Quantization has been proven to work well on convolutional neural networks and transformer-based models. Despite the decency of these models, recent works have shown that MLP-based models are able to achieve comparable results on various tasks ranging from computer vision, NLP to 3D point cloud, while achieving higher throughput due to the parallelism and network simplicity. However, as we show in the paper, directly applying quantization to MLP-based models will lead to significant accuracy degradation. Based on our analysis, two major issues account for the accuracy gap: 1) the range of activations in MLP-based models can be too large to quantize, and 2) specific components in the MLP-based models are sensitive to quantization. Consequently, we propose to 1) apply LayerNorm to control the quantization range of activations, 2) utilize bounded activation functions, 3) apply percentile quantization on activations, 4) use our improved module named multiple token-mixing MLPs, and 5) apply linear asymmetric quantizer for sensitive operations. Equipped with the abovementioned techniques, our Q-MLP models can achieve 79.68% accuracy on ImageNet with 8-bit uniform quantization (model size 30 MB) and 78.47% with 4-bit quantization (15 MB).
BibTeX
@misc{zhao2022analysis,
  author = {Lingran Zhao and Zhen Dong and Kurt Keutzer},
  title = {{Analysis of Quantization on MLP-based Vision Models}},
  year = {2022},
  eprint = {2209.06383},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.06383},
}
Figure