Introduction
Multi-Layer Perceptron (MLP) based vision models, such as MLP-Mixer and ResMLP, have emerged as efficient alternatives to Convolutional Neural Networks (CNNs) and Transformers. Their structural simplicity and high parallelism offer significant throughput advantages on modern hardware. However, deploying these models on edge devices typically requires quantization—converting 32-bit floating-point weights and activations into low-bit integers (e.g., 8-bit or 4-bit). Unlike CNNs, MLP-based models often experience catastrophic accuracy degradation when quantized. This paper investigates the root causes of this sensitivity and proposes a "quantization-friendly" toolkit to enable robust low-precision deployment.
The Quantization Challenge: Activation Outliers and Layer Sensitivity
The authors identify two primary obstacles to effective MLP quantization: extreme activation ranges and the inherent sensitivity of token-mixing layers.
1. Exploding Activation Ranges
Standard MLP architectures can produce activation values with extreme outliers. These outliers "stretch" the quantization grid, forcing the majority of critical data into a very narrow range of integer values, which leads to significant information loss. As shown in Figure 5, models like ResMLP and ConvMixer exhibit much higher maximum and top-1% activation values compared to ResNet or ViT.
2. Sensitive Token-Mixing Layers
Through Hessian trace analysis, the researchers found that token-mixing MLPs—which process spatial information across patches—are significantly more sensitive to quantization noise than channel-mixing layers. This sensitivity stems from the heavy reuse of parameters across all channels, where a small error in a shared weight can propagate across the entire feature map.
Proposed Solutions: The Q-MLP Toolkit
To address these issues, the authors introduce several architectural and algorithmic improvements:
- Restricting Activation Ranges: To stabilize activations, simple Affine layers are replaced with LayerNorm or BatchNorm. Additionally, unbounded activation functions (ReLU/GELU) are replaced with Parametrized Clipping Activation (PACT), which learns an optimal upper bound during training to clip outliers.
- Multi-Token-Mixing: To reduce the sensitivity of spatial processing, the authors propose dividing channels into groups and applying independent token-mixing MLPs to each group. This "multi-token-mixing" approach (illustrated in Figure 1) increases model expressivity and robustness without a substantial increase in parameter count.
- Advanced Quantization Schemes: The toolkit includes percentile quantization (clipping the top 1% of values) and asymmetric linear quantizers to better accommodate skewed activation distributions.
Key Results and Performance
The resulting Q-MLP models demonstrate that MLP-based architectures can indeed be quantized effectively:
- 8-bit Quantization: Achieved 79.68% accuracy on ImageNet-1k with a model size of only 30 MB.
- 4-bit Quantization: Maintained a competitive 78.47% accuracy (15 MB model size), significantly outperforming standard post-training quantization (PTQ) baselines.
- Hardware Efficiency: On a TITAN RTX GPU, the ResMLP-S24/16 variant achieved a throughput of 468 images/second, surpassing both ViT-S/16 (451 img/sec) and ResNet-50 (466 img/sec), validating the efficiency benefits of the MLP architecture.
Conclusion
By addressing the unique architectural sensitivities of MLP-based models, this work paves the way for their deployment in resource-constrained environments. The combination of activation range restriction and multi-token-mixing allows these models to retain their throughput advantages while achieving the memory efficiency of 4-bit and 8-bit quantization.
Summary updated