Quantizing deep neural networks to sub-byte precisions—such as 4-bit, 2-bit, ternary, or binary formats—drastically reduces memory footprints and energy consumption. However, running these ultra-low-precision models on commodity processors introduces a frustrating "hardware gap." As noted by Liu and Chen, standard Single Instruction, Multiple Data (SIMD) architectures define a minimum granularity, typically 8-bit or 16-bit integers. When executing sub-8-bit operations on these architectures, the remaining bits in the register lanes sit idle, wasting the hardware's computational capacity.
To bridge this gap, researchers have developed techniques to pack multiple sub-byte values into standard-width registers. By treating wide registers as parallel containers, both software kernels and specialized hardware extensions can achieve sub-word parallelism, unlocking massive throughput gains on hardware that lacks native low-precision support.
Software Emulation via Bit Shifting and Masking
On general-purpose edge CPUs, achieving sub-byte acceleration requires software-level strategies to map low-precision operations onto existing instruction sets. The most direct way to accomplish this is by packing multiple low-bit weights or indices into standard 8-bit or 32-bit containers.
Dong et al. and Dong et al. demonstrate that coupling multiple sub-8-bit weights into a single 8-bit slot allows commodity mobile processors to double their throughput. To execute arithmetic on these packed structures, custom SIMD kernels use bit-shifting and masking operations. As Lin et al. show, this approach bypasses the lack of native 4-bit hardware support, extracting and processing low-precision operands in parallel within standard integer slots.
For binary neural networks (BNNs), where weights and activations are represented by single bits, this extraction must be highly optimized. Zhang et al. utilize right-shift and overwrite assembly instructions to rapidly extract sign bits from full-precision tensors into a single vector. This approach avoids sequential comparisons and populates SIMD registers efficiently, accelerating downstream binary dot products.
Bypassing Arithmetic with SIMD Lookups
Instead of paying the computational cost of software bit-manipulation for arithmetic operations, an alternative strategy replaces arithmetic entirely with parallel table lookups executed directly inside SIMD registers.
Vector shuffle instructions (such as those found in x86 AVX-512 or ARM NEON) allow a processor to use the elements of one register as indices to select values from another register. Ashfaq leverages this mechanism by repurposing SIMD shuffle instructions as parallel lookup engines for ultra-low-precision (2-bit) matrix multiplication. Because the lookup table fits entirely within the processor's vector registers, the operation bypasses standard arithmetic units and runs faster than native 8-bit math.
Standard SIMD Lane (Wasted Space):
[ 4-bit Val | Idle Bits (4-bit) ] -> Processed as full 8-bit integer
Packed SIMD Lane (Sub-word Parallelism):
[ 4-bit Val A | 4-bit Val B ] -> Processed simultaneously via Shifting/Masking or Lookups
To scale this approach to higher-bit granularities, André and Le Scouarnec decompose larger lookup tables into smaller segments that fit within SIMD register limits. By utilizing vector blend operations to select the correct outputs from these segments, the system maintains high search accuracy while preserving the throughput of register-based parallel lookups.
Maximizing Density and Preventing Overflow
When packing multiple values into a single register lane, intermediate accumulations pose a significant challenge: they can easily overflow the narrow bit-width allocated to them. Upcasting these values to wider lanes (e.g., converting 8-bit integers to 16-bit lanes before addition) solves the overflow problem but halves the effective SIMD throughput.
To maintain maximum data density, Blalock et al. utilize SIMD averaging instructions for integer aggregation. Because averaging instructions inherently keep the result within the original 8-bit lane width, matrix operations can remain entirely within narrow registers without the overhead of upcasting. A closed-form bias correction is then applied downstream to restore numerical accuracy. Similarly, Ni et al. demonstrate that simulating low-precision accumulators via software-level bit-packing allows multiple low-bitwidth additions to run concurrently inside a single wide register.
At the architecture level, reducing the bit-width of the accumulators themselves can yield massive performance dividends. Xie et al. show that switching from 32-bit to 16-bit accumulators on mobile CPUs doubles inference speed by doubling the number of operations executed per clock cycle. Crucially, this transition minimizes data transfers between CPU registers and main memory—a primary bottleneck in edge deployment.
Hardware-Assisted Slicing and Native Sub-Byte SIMD
While software emulation makes sub-byte quantization viable on commodity hardware, the bit-shifting, masking, and shuffling required to unpack data introduces an "unpacking tax" that consumes clock cycles and energy. To eliminate this overhead, recent architectural designs integrate data manipulation directly into the hardware logic.
Ottavi and Nadalini and Rutishauser propose integrating specialized hardware slicing and routing units directly into the processor's dot-product and arithmetic logic units. These hardware units automatically extract, sign-extend, and align low-precision operands from wider memory words within a single clock cycle. This architectural shift completely removes the need for software-based bit-manipulation.
Furthermore, to prevent instruction set bloat when dealing with multiple sub-byte formats, Ottavi decouples precision metadata from the instruction set by utilizing status registers, allowing a single opcode to handle varying bit-widths flexibly. When paired with native hardware support for sub-byte SIMD operations—such as the 2-bit, 4-bit, and ternary extensions explored by Garofalo et al. and Rutishauser et al.—throughput scales near-linearly as bit-width decreases. This native support allows spatial hardware accelerators to maximize their computational density by repurposing high-precision units into dense arrays of parallel, low-precision engines (Tagliavini et al.; Zhou et al.).
Key Foundational Work
A Transprecision Floating-Point Platform for Ultra-Low Power Computing (2017) — contributed 1 concept elements to this direction.
[DL] A Survey of FPGA-based Neural Network Inference Accelerators (2019) — contributed 1 concept elements to this direction.
daBNN: A Super Fast Inference Framework for Binary Neural Networks on ARM devices (2019) — contributed 1 concept elements to this direction.