Distribution-Aware Hadamard Quantization for Hardware-Efficient Implicit Neural Representations

Wenyong Zhou, Jiachen Ren, Taiqiang Wu, Yuxin Cheng, Zhengwu Liu, Ngai Wong

Abstract

Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing continuous signals such as images and 3D scenes. However, their reliance on complex activation functions and high-precision floating-point arithmetic poses significant challenges for hardware deployment. We propose Distribution-Aware Hadamard Quantization (DHQ), a framework that leverages the Hadamard transform to standardize diverse data distributions into a hardware-friendly Gaussian-like shape. Our approach enables full 8-bit quantization of both weights and activations. Implementation on an FPGA reveals a 98.3% reduction in LUT utilization and a 32.7% improvement in latency while maintaining high reconstruction fidelity.

Introduction

Implicit Neural Representations (INRs) utilize Multi-Layer Perceptrons (MLPs) to map coordinates to signal values, enabling continuous and high-resolution representations. Despite their success, INRs incur a heavy "hardware tax." To capture fine details, models like SIREN employ periodic activation functions (e.g., sin), which typically necessitate 32-bit precision. While prior work has explored weight quantization, activations often remain at high precision, leading to bulky and power-hungry Multiply-Accumulate (MAC) units on hardware like FPGAs. The primary obstacle to full quantization is the erratic nature of INR data distributions, which vary significantly across layers and often exhibit non-standard shapes that are difficult to quantize efficiently.

Methodology: Distribution-Aware Hadamard Quantization

The DHQ framework addresses distribution inconsistency by introducing a standardization step prior to quantization.

Distribution Analysis and Standardization

Layer-wise distributions in INRs are highly inconsistent; for instance, early-layer activations in SIREN models are often U-shaped, whereas final output layers may appear bell-shaped. Rather than designing specialized quantizers for each unique distribution, DHQ applies a Hadamard transformation to both weights and activations.

The Hadamard transform is a linear, orthogonal operation that redistributes data energy. According to the Central Limit Theorem, the transformed elements—which become sums of many independent contributions—converge toward a unified, Gaussian-like bell shape.

Figure 6: Histogram of activation values after Hadamard transformation, showing the resulting bell-shaped distribution.

Hardware-Efficient Quantization

Once standardized into a bell shape, the data can be effectively mapped using a simple, hardware-friendly uniform 8-bit quantizer. This uniformity across layers significantly simplifies the hardware architecture. The researchers implemented this on a specialized FPGA accelerator featuring 256 parallel MAC arrays and a pipelined architecture optimized for 8-bit fixed-point operations.

Experimental Results

The DHQ method was evaluated on image reconstruction tasks using natural images (Cameraman, Kodak02) and biomedical CT scans.

Hardware Utilization and Performance

The transition from 32-bit (W32A32) to 8-bit (W8A8) precision via DHQ yields massive resource savings. As shown in Figure 1, LUT utilization is reduced by 98.3%, and DSP usage drops by 60%.

Figure 1: Comparison of FPGA resource utilization across different quantization levels. The W8A8 configuration enabled by DHQ drastically reduces resource requirements.

Furthermore, the 8-bit implementation achieved a 32.7% reduction in latency and a 40.1% decrease in energy consumption compared to the baseline.

Reconstruction Quality

DHQ maintains high visual fidelity. On the SIREN model, DHQ achieved a PSNR of 31.57 dB, closely matching the full-precision performance of 32.13 dB and significantly outperforming standard uniform 8-bit quantization (29.65 dB). Qualitatively, DHQ preserves critical textures and boundaries that are often lost or corrupted by noise in traditional quantization schemes.

Conclusion and Future Work

DHQ demonstrates that reshaping data distributions via the Hadamard transform is a highly effective strategy for quantizing INRs. By enabling 8-bit activation quantization, it unlocks substantial hardware efficiency gains that weight-only quantization cannot provide. Future research will explore the applicability of DHQ to 3D representations like NeRFs and investigate the feasibility of even lower bit-widths, such as 4-bit or 2-bit quantization.

Summary updated

BibTeX
@inproceedings{zhou2025distribution,
  author = {Wenyong Zhou and Jiachen Ren and Taiqiang Wu and Yuxin Cheng and Zhengwu Liu and Ngai Wong},
  title = {{Distribution-Aware Hadamard Quantization for Hardware-Efficient Implicit Neural Representations}},
  booktitle = {ICME},
  year = {2025},
  doi = {10.1109/icme59968.2025.11210085},
}
Figure