Abstract
Mixed-precision quantization is essential for deploying deep neural networks on resource-constrained edge devices, yet finding the optimal bit-width for each layer remains a significant computational bottleneck. We present OMPQ (Orthogonal Mixed Precision Quantization), a highly efficient method that replaces iterative search algorithms with a geometric approach. By introducing the Orthogonality Metric (ORM) as a proxy for model accuracy, OMPQ transforms bit-allocation into a linear programming problem. Our method requires zero search iterations and as few as 32 images to achieve state-of-the-art results, reducing search time from hours to seconds.
Introduction
To run deep learning models on edge devices like smartphones or IoT sensors, quantization—reducing the precision of weights and activations from 32-bit floats to low-bit integers—is a standard practice. Mixed-precision quantization is particularly effective because it allows sensitive layers to maintain higher precision while aggressively compressing less critical ones.
However, the search space for bit configurations is exponentially large. Existing solutions typically rely on reinforcement learning, evolutionary algorithms, or second-order Hessian information. These methods are computationally expensive, often requiring hundreds of iterations, substantial training data, and significant GPU time. OMPQ addresses this "search bottleneck" by providing a deterministic, data-efficient alternative.
Methodology
OMPQ shifts the quantization problem from an iterative search to a geometric optimization. The core of the approach is the Orthogonality Hypothesis: a layer's mathematical orthogonality—the independence of its representation relative to others—is a strong indicator of its sensitivity to precision loss.
The OMPQ Pipeline
- Network Deconstruction: The neural network is viewed as a sequence of functions. The method analyzes the representational capability of these functions using a small batch of data.
- ORM Calculation: Calculating true orthogonality is mathematically complex. OMPQ uses Monte Carlo sampling and a Frobenius norm-based acceleration trick to approximate the Orthogonality Metric (ORM) in a single forward pass.
- Linear Programming (LP): Bit allocation is formulated as a constrained optimization problem. The objective is to maximize the total weighted orthogonality of the network while staying within a specific model size budget. This LP problem is solved almost instantaneously on a standard CPU.
Key Insights
- Orthogonality as an Accuracy Proxy: The researchers identified a strong positive correlation between a model's total ORM and its Top-1 accuracy. This allows the ORM to serve as a reliable stand-in for accuracy during the configuration phase, eliminating the need for expensive validation cycles.
- Efficiency and Data Sparsity: Unlike prior methods that may require the entire training set or over 1,000 images, OMPQ achieves peak performance using only 32 to 64 images.
- Layer Sensitivity: Analysis using ORM revealed that $3 \times 3$ convolutional layers are generally more orthogonal than $1 \times 1$ layers. This explains why architectures like MobileNetV2, which rely heavily on $1 \times 1$ convolutions, are notoriously difficult to quantize and benefit most from OMPQ's precise bit allocation.
Experimental Results
We evaluated OMPQ on the ImageNet (ILSVRC12) dataset across ResNet-18, ResNet-50, and MobileNetV2 architectures.
Performance and Speed
- Search Speed: For MobileNetV2, the bit-width search takes only 9 seconds on a CPU, whereas reinforcement learning methods can take hours or even days.
- Accuracy: On ResNet-18, OMPQ achieved 72.08% Top-1 accuracy at a 6.7Mb model size, outperforming HAWQ-V3 (70.22%). On MobileNetV2, OMPQ reached 71.39% accuracy at 1.5Mb, significantly surpassing the BRECQ baseline of 70.28%.
As illustrated above, OMPQ reduces the "Searching Data" and "Searching Iterations" by orders of magnitude compared to established baselines, effectively achieving a "zero-iteration" search.
Conclusion and Future Work
OMPQ demonstrates that geometric properties of neural networks can be leveraged to bypass expensive search processes in model compression. While currently focused on CNNs, the principles of orthogonality could potentially be applied to Transformers and Large Language Models (LLMs). Future research will explore more complex optimization formulations, such as treating bit-allocation as a "multiple knapsack problem" to further refine the trade-offs between hardware constraints and model performance.
Summary updated