ApproxTorch 2.0: A High Performance Simulation Framework of 8-bit Signed Integer Approximate Multipliers for Convolutional Neural Networks

Ke Ma, Shinichi Nishizawa, Shinji KimuraIEICE 2025 · DOI

This paper introduces ApproxTorch 2.0, a high-performance simulation framework designed to bridge the gap between hardware-level approximate computing and software-level deep learning. By integrating custom CUDA kernels into PyTorch, the authors enable researchers to simulate the effects of "imperfect" hardware multipliers on Convolutional Neural Networks (CNNs) at unprecedented speeds.

TL;DR

ApproxTorch 2.0 is a GPU-accelerated framework that allows developers to simulate and retrain CNNs using 8-bit signed integer approximate multipliers. By utilizing custom CUDA kernels and Look-Up Tables (LUTs), it achieves up to a 171× speedup over CPU simulations, making it feasible to evaluate hardware-inspired arithmetic errors on large-scale datasets like ImageNet.

What Problem Does This Paper Address?

In the quest for energy-efficient AI, hardware designers often use Approximate Computing. This involves designing multipliers that are smaller and faster but occasionally produce incorrect results. While CNNs are famously robust to noise, evaluating exactly how much error a specific hardware design introduces requires simulating millions of operations with that specific error profile.

The challenge is that standard deep learning frameworks (like PyTorch or TensorFlow) are optimized for exact floating-point math. Simulating "broken" 8-bit integer math usually requires slow, custom C++ code or MATLAB scripts. Previous attempts to integrate this into PyTorch were either too slow for large models or lacked support for "retraining"—the process of teaching a model to compensate for its hardware's mathematical quirks.

Methods

The authors developed a system that mimics 8-bit signed integer arithmetic within the PyTorch ecosystem using several key techniques:

  1. Look-Up Table (LUT) Simulation: Instead of performing a standard multiplication, the system uses the two 8-bit inputs as indices to look up a pre-calculated result in a 256KB table. This table can contain the exact results or the "erroneous" results of an approximate hardware multiplier.
  2. Custom CUDA GEMM Kernels: Standard GPU libraries (like cuBLAS) don't support LUT-based multiplication. The authors wrote a dedicated CUDA kernel for General Matrix Multiplication (GEMM). They optimized this using block tiling (moving data into fast shared memory) and thread tiling (using registers) to overcome the memory bandwidth bottleneck caused by frequent LUT access.
  3. Straight-Through Estimator (STE): To allow the model to learn, the authors implemented STE. During the "forward pass," the model uses the approximate, error-prone math. During the "backward pass" (training), it treats the gradients as if the math were exact, allowing the optimizer to update weights effectively despite the non-differentiable nature of discrete integer errors.
  4. Fake Quantization: The framework uses 32-bit floats to "carry" 8-bit integer values, ensuring compatibility with existing PyTorch layers while mimicking the precision loss of quantization.

Datasets

The framework was validated across a range of standard benchmarks:

  • MNIST: Using the LeNet5 architecture.
  • CIFAR-10: Using ResNet20, ResNet32, ResNet50, VGG16, and VGG19.
  • ImageNet: Using ResNet18, ResNet34, and ResNet50. This is significant because ImageNet is often too computationally expensive for previous approximate simulation tools.

Key Insights and Ideas

  • The Memory Bandwidth Wall: The authors discovered that the primary bottleneck isn't the calculation itself, but the memory reads. Because a 256KB LUT is too large for the fastest "shared memory" on most GPUs (<100KB), they had to optimize how the matrices were cached to leave enough bandwidth for the LUT lookups in global memory.
  • Retraining is a Game Changer: The paper shows that some approximate multipliers can tank a model's accuracy (e.g., dropping a ResNet from 92% to 17%). However, using their STE-based retraining, they could recover that accuracy almost entirely (back to 92%+).
  • 4-bit Efficiency: In an appendix, the authors show that 4-bit multipliers are even faster to simulate because their much smaller LUT (16KB) can fit entirely in the GPU's shared memory, providing an additional 14% speedup.

Key Results

  • Speedup: ApproxTorch 2.0 is 171× faster than CPU-based simulation and 17.2× faster than the previous version (ApproxTorch 1.0) when running ResNet50.
  • Inference Overhead: The "Normalized Time" (the ratio of approximate simulation time to native exact math time) was reduced to between 2.5× and 6.5×. For context, previous state-of-the-art tools were often 11× slower or more.
  • Accuracy Recovery: For a variety of approximate multipliers from the EvoApprox8b library, the framework successfully retrained models to within 0.5% of their original baseline accuracy.

Limitations and Future Work

  • Hardware Tuning: The performance of the custom CUDA kernels is highly dependent on "tiling parameters" (BM, BN, BK, etc.), which currently need to be manually tuned for different GPU architectures (e.g., RTX A6000 vs. V100).
  • Layer Constraints: The current implementation focuses on 2D Convolution and Linear layers. While these represent the bulk of CNN computations, other layers (like Depthwise Convolutions or Attention mechanisms in Transformers) are not yet explicitly optimized.
  • Future Directions: The authors suggest expanding the framework to support more diverse model types and exploring even lower-bit precisions where the entire simulation can reside in high-speed on-chip memory.

Summary updated

Abstract

Recent advancements in Convolutional Neural Networks (CNNs) have led to significant increases in computational complexity and power consumption. Approximate computing, especially through the use of approximate multipliers, has emerged as a promising approach to realize the reduction of power, area and delay by sacrificing some computational precision. Approximate multipliers produce results with errors, thus their effects to CNN accuracies need to be evaluated. However, the lack of efficient software tools for simulating approximate multipliers hinders their widespread adoption. This paper proposes ApproxTorch 2.0, a high perfomance simulation framework based on PyTorch and CUDA for evaluating 8-bit signed integer based approximate multipliers for CNNs. To realize higher performance, CUDA and Nvidia GPUs are used to accelerate the simulation process. Two approximate layers are implemented: 2D Convolution layer and Linear layer. The simulation of approximate multiplications is done by accessing pre-defined look-up tables (LUTs) stored in GPU memory. By dedicated optimization in the customized CUDA approximate GEMM kernel, ApproxTorch 2.0 achieves 171× speedup compared with CPU simulation and 17.2× speedup compared with previous ApproxTorch 1.0 on running ResNet50. Lastly, gradient estimation is added to support the retraining of approximate CNNs with quantized values and proved to be effective for recovering the accuracy loss after approximation.
BibTeX
@inproceedings{ma2025approxtorch,
  author = {Ke Ma and Shinichi Nishizawa and Shinji Kimura},
  title = {{ApproxTorch 2.0: A High Performance Simulation Framework of 8-bit Signed Integer Approximate Multipliers for Convolutional Neural Networks}},
  booktitle = {IEICE},
  year = {2025},
  doi = {10.1587/transfun.2025eap1070},
}
Figure