Quantizing a neural network reduces its memory footprint and accelerates inference by replacing high-bitwidth floating-point numbers (like FP32) with lower-bitwidth representations (like INT4 or INT8). However, assigning a uniform bit-width across an entire network is rarely optimal. Some layers are highly sensitive to quantization noise and degrade drastically if compressed too aggressively, while other layers are highly resilient.
Mixed-Precision Quantization (MPQ) solves this by assigning a custom bit-width to each layer or component. But finding the optimal configuration is a massive combinatorial challenge. If a network has $L$ layers and we can choose from $B$ possible bit-widths, there are $B^L$ candidate configurations. Searching this exponential space using traditional methods like Reinforcement Learning (RL) or Neural Architecture Search (NAS) requires training or evaluating thousands of configurations, demanding hours or days of compute.
The breakthrough insight across recent literature is that we can bypass iterative searching entirely. By treating the network's layers as decoupled components, the global bit-allocation problem maps perfectly onto a classic mathematical framework: the Knapsack Problem. This allows researchers to use deterministic Integer Linear Programming (ILP) or standard Linear Programming (LP) solvers to find the globally optimal precision policy in seconds.
The Core Intuition: Decoupling the Network
To transform an exponential search space into a solvable linear program, we must be able to evaluate the "cost" and "benefit" of quantizing each layer independently.
Imagine a network where we need to choose a bit-width for each layer. We want to minimize the total accuracy loss (or maximize the total model robustness) while keeping the total model size under a hardware memory budget. If we can approximate the total model error as the sum of individual layer errors, the problem becomes linear.
Instead of searching through combinations, the optimization problem shifts to measuring layer properties. Once we compute a sensitivity metric for each layer across the available bit-widths, an ILP solver can instantly navigate the trade-offs to find the optimal configuration along the efficiency-performance Pareto frontier, as noted by Lee et al..
Iterative Search (RL/NAS):
[Try Policy A] -> [Evaluate Model (Slow)] -> [Update Search] -> [Try Policy B] ...
Analytical Optimization (ILP):
[Single-Pass Sensitivity Measurement] -> [ILP Solver (Instant)] -> [Optimal Policy]
Step 1: Measuring Layer Sensitivity Without Training
To construct the objective function for a linear solver, we need a fast proxy for how a layer's bit-width affects the global model. Researchers have proposed several distinct metrics that can be computed in a single forward pass or directly from the model's structural properties:
- Bit Gradients: Kundu 0002 et al. utilize bit-gradients during training from scratch to track how sensitive the global loss function is to precision changes in each layer.
- Information Retrieval Analogies: Wang et al. adapt the TF-IDF metric from natural language processing to mixed-precision quantization. By treating discriminative features as "words" and layers as "documents," they measure how well individual layers preserve class separability.
- Geometric and Structural Metrics: Ma et al. introduce an orthogonality metric to serve as an accuracy proxy, while Li et al. leverage Topological Entropy to measure a layer's sensitivity to bit reduction in a single pass.
- Learnable Parameters: Tang et al. repurpose the learnable scale factors inherent to quantization-aware training, treating them directly as indicators of layer importance.
Step 2: Formulating the Knapsack Problem
Once the sensitivity scores are calculated, the bit-width selection is cast as a discrete optimization problem. Let $x_{i,b}$ be a binary choice variable that equals $1$ if layer $i$ is assigned bit-width $b$, and $0$ otherwise.
If $S_{i,b}$ represents the sensitivity or importance score of layer $i$ at bit-width $b$, and $C_{i,b}$ represents its hardware cost (e.g., memory footprint or bit operations), the optimization problem is formulated as:
$$\max \sum_{i} \sum_{b} S_{i,b} x_{i,b}$$
$$\text{subject to } \sum_{i} \sum_{b} C_{i,b} x_{i,b} \le \text{Budget}$$
$$\text{and } \sum_{b} x_{i,b} = 1 \quad \forall i$$
The final constraint ensures that each layer is assigned exactly one bit-width. As Yang et al. demonstrate, this formulation guarantees that the chosen configuration strictly adheres to physical hardware limits while minimizing accuracy loss.
Although general integer linear programming is NP-hard, the number of layers in a typical neural network (dozens to hundreds) is small enough that commercial solvers find the exact, globally optimal solution in milliseconds. For example, Tang et al. demonstrate that an optimal policy for a ResNet18 architecture can be discovered in just 0.06 seconds.
Adapting to Complex Constraints and Modern Architectures
As deep learning models have evolved beyond standard convolutional networks, the constrained optimization paradigm has adapted to handle multi-dimensional hardware constraints and non-linear layer interactions.
Hardware-Latency Co-Design
Memory footprint is not the only hardware constraint; execution latency on real hardware is often non-linear with respect to bit-width. Duanmu et al. incorporate actual GPU execution latency alongside memory budgets into an ILP formulation for Mixture-of-Experts (MoE) models, ensuring that precision allocation translates directly to actual throughput gains.
Accounting for Layer Synergy
Layers do not always operate in complete isolation. To address this, Ranjan et al. extend the optimization framework from linear programming to Integer Quadratic Programming. This allows the solver to account for "synergy" and joint dependencies between neighboring layers when quantizing large vision models like the Segment Anything Model (SAM).
Beyond Standard Layers
The strategy of mapping precision allocation to discrete optimization solvers has proven highly versatile across diverse domains:
- Diffusion Models: Zhao et al. use integer programming to decouple image quality and text alignment metrics, optimizing layer-wise precision for few-step text-to-image diffusion models under strict memory budgets.
- LLM Training: Rather than focusing solely on post-training quantization, Pan et al. apply an ILP knapsack formulation to dynamically adjust layer-wise precision during the training of Large Language Models, balancing computational efficiency against training stability.
- Weight Decomposition: Xiong et al. apply ILP to allocate bit-widths to singular vectors during SVD-based delta compression for LLMs, minimizing reconstruction errors under a fixed bit budget.
- Modular Architectures: Javed et al. utilize ILP to solve for the optimal distribution of bit-widths across distinct structural modules in 6D object pose estimation networks.
- Mixture-of-Experts (MoE): Huang et al. employ linear programming to distribute discrete bit-widths across components in MoE LLMs, ensuring influential parameters retain higher precision while meeting a target average bit-width.
- Physics Simulations: The paradigm even extends outside of deep learning. Liu et al. apply constrained optimization via Lagrange multipliers to automatically determine the minimum precision required for individual variables in physics-based simulations to satisfy a global error bound.
Key Foundational Work
CSMPQ: Class Separability Based Mixed-Precision Quantization (2022) — contributed 2 concept elements to this direction.
BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratch (2022) — contributed 2 concept elements to this direction.
Mixed-Precision Neural Network Quantization via Learned Layer-Wise Importance (2022) — contributed 1 concept elements to this direction.