TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precision

Jinhee Kim, Seoyeon Yoon, Taeho Lee, Joo Chan Lee, Kang Eun Jeon, Jong Hwan Ko

Introduction

Deploying Deep Neural Networks (DNNs) on edge devices—such as smartphones, IoT sensors, and wearables—requires a delicate balance between computational accuracy and hardware efficiency. In dynamic environments, the optimal bit-precision for a model may change based on battery levels or real-time performance requirements. While existing techniques like "Once-For-All" (OFA) or "Any-Precision" networks allow for variable bit-widths, they often rely on keeping a high-precision floating-point (FP32) "parent" model in memory to re-quantize weights for lower-precision modes. This requirement significantly increases storage and memory overhead, undermining the efficiency goals of edge deployment.

TruncQuant addresses this challenge by introducing a training scheme that makes models "truncation-ready." By aligning quantization boundaries with truncation logic during training, TruncQuant allows a single model to switch between bit-precisions (e.g., from 8-bit to 2-bit) using simple bit-shifting operations, eliminating the need for a parent model and drastically reducing memory footprints.

The Problem: The Quantization-Truncation (QT) Gap

The most efficient way to reduce the precision of an integer is truncation—simply dropping the least significant bits (LSBs). However, standard Quantization-Aware Training (QAT) typically employs rounding functions to map floating-point weights to integers. This creates a fundamental mathematical mismatch known as the Quantization-Truncation (QT) Gap.

When a model is trained using standard 8-bit rounding and then truncated to 2-bit at runtime, the accuracy often collapses. For instance, a standard ResNet-50 model can experience a 67.74% drop in accuracy when subjected to naive truncation. This occurs because the mathematical boundaries of rounding do not align with the boundaries of bit-shifting; the weights that the network learned to interpret as specific values are effectively "misaligned" when the LSBs are removed.

The TruncQuant Methodology

TruncQuant redesigns the quantization process to ensure that the Most Significant Bits (MSBs) remain consistent across different precision levels. The methodology consists of three core components:

1. Floor-Based Quantization

Instead of standard rounding, TruncQuant utilizes a floor-based quantization function: $$q_t(W; n) = \lfloor (M_n + 1)W \rfloor$$ This function ensures that the "bins" (the ranges of floating-point values mapping to a specific integer) are evenly distributed and follow a multiplicative relationship. By structuring the bins this way, the integer representation of a weight at a lower precision is a direct prefix of its representation at a higher precision.

2. MSB Consistency and Bit-Shifting

Because the bin widths are mathematically aligned, the MSBs of a weight remain identical whether the model is operating at 8-bit or 2-bit precision. At runtime, the system only needs to store the highest-precision version of the weights (e.g., INT8). To switch to a lower precision, such as 4-bit, the hardware simply performs a bitwise right-shift. This eliminates the need to store or access the original FP32 weights.

3. Modified Straight-Through Estimator (STE)

To maintain theoretical consistency during backpropagation, the authors adjusted the gradient scaling. Since the bin widths in TruncQuant are $1/(M_n + 1)$ rather than the standard $1/M_n$, the gradient updates are scaled accordingly to ensure stable training and convergence.

Experimental Results

The researchers validated TruncQuant across various benchmarks, including CIFAR-10, SVHN, and ImageNet, using architectures such as ResNet-18, ResNet-50, MobileNetV2, and AlexNet.

Accuracy and Recovery

On the ImageNet dataset with ResNet-50, TruncQuant demonstrated remarkable robustness. While naive truncation of a standard model to 2-bits results in near-zero accuracy, TruncQuant recovers this performance almost entirely. The resulting accuracy is on par with models specifically trained for a single, fixed bit-width, proving that flexibility does not have to come at the cost of performance.

Storage and Energy Efficiency

By making the quantized weights themselves truncatable, TruncQuant eliminates the "parent model" storage requirement. This leads to a 3.53× storage saving compared to previous state-of-the-art flexible precision methods. Furthermore, hardware simulations using the Timeloop/Eyeriss framework showed significant energy savings. Because the model avoids fetching FP32 weights from DRAM during precision switching, it bypasses one of the most energy-intensive operations in edge computing.

Discussion and Limitations

TruncQuant serves as a versatile "plug-in" strategy that can be integrated with existing frameworks like RobustQuant and EQ-Net. However, some limitations remain:

  • 1-Bit Support: The current floor-based logic requires a minimum range of integer values to function, meaning 1-bit (binary) quantization is not yet supported.
  • Layer Sensitivity: In line with common practices, the first and last layers of the networks were not quantized to ultra-low bits, as these layers are critical for maintaining overall model stability.
  • Hardware Realization: While energy benefits were demonstrated via high-fidelity simulations, future work is needed to measure real-world power savings on physical FPGA or ASIC accelerators.

Conclusion

TruncQuant provides a mathematically elegant solution to the QT Gap, enabling seamless, on-the-fly precision adjustments for DNNs. By aligning training logic with hardware-friendly bit-shifting, it offers a path toward truly efficient, adaptive AI for the next generation of edge devices.

Summary updated

BibTeX
@inproceedings{kim2025truncquant,
  author = {Jinhee Kim and Seoyeon Yoon and Taeho Lee and Joo Chan Lee and Kang Eun Jeon and Jong Hwan Ko},
  title = {{TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precision}},
  booktitle = {ISLPED},
  year = {2025},
  doi = {10.1109/islped65674.2025.11261696},
}
Figure