Deep neural networks are increasingly deployed in environments with highly volatile resource constraints. An edge device may experience fluctuating battery levels and available memory bandwidth, while a cloud server hosting a Large Language Model (LLM) must constantly adapt to sudden spikes in user traffic.
To maintain efficiency, systems must dynamically adjust model precision—dropping to ultra-low bit-widths (like 2-bit or 4-bit) during resource scarcity and scaling up to higher precision (like 8-bit or 16-bit) when accuracy is paramount.
Traditionally, achieving this flexibility required a massive storage penalty. As noted by Yang et al. in AdaBits: Neural Network Quantization With Adaptive Bit-Widths, maintaining separate, specialized weight tensors for every supported precision level creates substantial storage and memory overhead on edge devices. Swapping these independent models in and out of memory also introduces severe latency bottlenecks.
The core insight of nested bit representation (or nested quantization) is that we can eliminate this redundancy by making lower-precision weights a physical subset of higher-precision weights. Instead of storing multiple models, a system stores a single, unified set of weights that can be instantly truncated or expanded at runtime to match changing hardware constraints.
The Mechanism of Nested Bit-Slicing
How can a single parameter represent multiple precisions simultaneously? The solution treats quantization as a hierarchical bit-stacking problem, structurally resembling a Matryoshka doll.
In a nested weight representation, a high-precision value is decomposed into a high-order base component containing the Most Significant Bits (MSBs) and one or more low-order refinement components containing the Least Significant Bits (LSBs). As explained by Hai et al. in Vertical Layering of Quantized Neural Networks for Heterogeneous Inference, defining weights through a recursive bit-stacking rule allows higher-precision models to physically encapsulate their lower-precision counterparts.
Standard 4-bit Integer: [ b3 | b2 | b1 | b0 ]
\________/\________/
Base MSB Refinement LSB
(2-bit) (Incremental)
To make this execution seamless, the underlying quantization scheme must satisfy a strict prefix property: the $k$-bit representation of a value must be an exact prefix of its $(k+1)$-bit representation. Park et al. leverage this property in Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs to allow a system to dynamically adjust precision by simply truncating or appending bits on the fly.
When a device enters a high-efficiency mode, it drops the refinement bits and executes inference using only the base MSBs. When high-fidelity performance is required, the system loads the residual bitplanes and appends them back to the base. Choi et al. demonstrate in Dual-Precision Deep Neural Network that this mechanism enables instantaneous transitions between accuracy and efficiency modes via simple bitwise operations, completely eliminating the latency and memory overhead of loading separate model files from disk.
This structural decomposition applies to floating-point representations as well. For example, Lee et al. introduce NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs, which decomposes standard 16-bit floating-point values (FP16) into two distinct 8-bit components representing the upper and lower bits. This allows a single memory footprint to support both reduced-precision (FP8) and full-precision execution without storing redundant model copies.
Aligning Quantization Boundaries for Truncation
Why can we not simply chop off the lower bits of a standard quantized model? If you take a standard INT8 model and truncate the last 4 bits, its accuracy typically collapses to near-zero. This failure occurs because standard quantization functions do not align their discrete decision boundaries with bitwise truncation logic.
To make a model "truncation-ready," the quantization function must be mathematically constrained so that the core information remains intact when lower-order bits are discarded.
Bulat et al. address this in Bit-Mixer: Mixed-precision networks with runtime bit-width selection by designing quantization functions with specific floor rounding operations and power-of-two scaling. This mathematical constraint guarantees that lower-bit representations remain exact, functional subsets of higher-bit ones.
Similarly, Kim et al. demonstrate in TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precision that structuring quantization bins to follow a strict multiplicative relationship ensures that the most significant bits of a weight remain identical across different bit-precisions. By aligning these boundaries during training, the network's weights naturally preserve their core representational meaning under simple bit-shifting operations.
Mitigating Bit-Width Interference Through Co-Training
Forcing a single set of weights to operate effectively across multiple bit-widths introduces a fundamental optimization challenge known as bit-width interference. If you optimize a weight tensor purely to maximize 2-bit accuracy, you may severely degrade its capacity to utilize the extra resolution available at 8-bit precision.
To resolve this conflict, models are optimized using joint multi-precision training objectives. The authors of Matryoshka Quantization introduce a joint objective that aggregates losses from several different quantization levels simultaneously during training:
$$\mathcal{L}_{\text{total}} = \sum_{b \in B} \alpha_b \mathcal{L}_b(W_{\text{truncated to } b})$$
This multi-scale objective forces the network to learn a unified representation where the primary bits (MSBs) are highly effective for low-precision inference on their own, while the additional refinement bits act as precise error-correction terms.
Furthermore, Tang et al. show in Retraining-free Model Quantization via One-Shot Weight-Coupling Learning that coupling weights across different precision levels during a single training phase allows for the seamless extraction of various mixed-precision sub-networks. This co-training paradigm facilitates bidirectional knowledge transfer between precision levels. As a result, the constraints of bit-sharing do not inherently degrade the representational capacity of the network at any individual precision level (Choi et al.).
Granular Runtime Adaptation
Once a model's weights are structured in a nested format, systems can vary precision at an incredibly fine granularity—moving far beyond static, model-wide quantization choices.
Instance-Dependent Quantization
Instead of running an entire model at a fixed precision, a network can adapt its bit-width on an input-by-input basis. Liu et al. demonstrate this in Instance-Aware Dynamic Neural Network Quantization, where a lightweight controller assigns lower precision to simple, easily classifiable inputs and higher precision to complex inputs. Because the lower-precision weights are generated dynamically from the single high-precision weight set via truncation, the model's total memory footprint never exceeds that of a standard static model.
Phase-Dependent LLM Decoding
In autoregressive LLM generation, the computational and error characteristics shift dramatically between the initial prompt processing phase (prefill) and the subsequent token generation phase (decoding).
Chen et al. exploit this variance in Progressive Mixed-Precision Decoding for Efficient LLM Inference by applying higher precision during the error-sensitive prefill phase and early decoding stages. As generation continues and the model becomes more resilient to quantization errors, the system progressively lowers the weight precision by dropping bitplanes. This progressive truncation maximizes token throughput while preserving the overall semantic quality of the output.
Key Foundational Work
Dual Precision Deep Neural Network (2020) — contributed 4 concept elements to this direction.
AdaBits: Neural Network Quantization With Adaptive Bit-Widths (2020) — contributed 3 concept elements to this direction.
Cascade^CNN: Pushing the Performance Limits of Quantisation in Convolutional Neural Networks (2018) — contributed 1 concept elements to this direction.