Realizing Low-Precision Storage and Sparse Execution in a Folded FPGA Speech Enhancer

Clément LarochearXiv 2026 · arXiv

TL;DR

This paper demonstrates that for "folded" FPGA accelerators—where a single compute engine is reused across multiple layers—standard hardware-aware training metrics like "bit-operations" are misleading. By specifically targeting weight storage and structured channel sparsity during training, the authors achieved a 60% reduction in dynamic energy and a 45% speedup on a Kria KV260, while actually improving speech quality compared to dense models.

What Problem Does This Paper Address?

Speech enhancement models for hearing aids and wearables need to be fast, private, and extremely energy-efficient. While FPGAs are ideal for this, a "spatial" implementation (where every layer of a neural network gets its own dedicated hardware) often consumes more resources than a small device can provide.

The solution is folding: reusing one physical compute engine to process the network's blocks in sequence. However, most "hardware-aware" training methods optimize for proxy metrics like the total number of bit-operations. In a folded architecture, these proxies fail. For example, if your hardware engine uses a fixed 8-bit multiplier, reducing a specific layer's weights to 4 bits saves storage space but doesn't necessarily speed up the computation. To truly optimize a folded engine, training must focus on two specific hardware costs: on-chip storage (how many bits are packed into memory) and scheduled cycles (how many operations can be skipped entirely).

Methods

The authors implement Conv-FSENet, a dilated temporal convolutional network (TCN), on an AMD Kria KV260 FPGA.

Fig. 1. Conv-FSENet Network. The accelerator returns 257 logits, and the host applies σ.

Key technical components include:

  • The Folded Engine: Instead of nine separate TCN blocks, they built one engine that fetches block-specific weights and history at each iteration.
  • HGQ-HLS4ML Flow: They used High Granularity Quantization (HGQ) to learn optimal bit-widths for every tensor and channel.
  • Engine-Aware Training: They modified the training loss function to include terms for the actual "tape" size (weight storage) and the number of MAC (multiply-accumulate) cycles.
  • Structured Sparsity: They applied group-lasso regularization to drive entire output channels to zero. In a folded engine, if a channel is exactly zero, the hardware controller can skip that entire loop in the schedule, directly translating sparsity into lower latency.
The folded architecture where one block engine is reused sequentially.

Datasets

The models were trained and evaluated using the 16 kHz VoiceBank-DEMAND dataset, a standard benchmark for speech enhancement. The evaluation used a speaker-disjoint test set of 824 utterances to ensure the model generalizes to new voices.

Key Insights and Ideas

  • Sparsity is the Latency Killer: In a folded design, reducing bit-width saves power and storage but not time. Only structured sparsity—removing entire rows from the MAC schedule—meaningfully shortens the execution time.
  • The "Aha" Moment on Regularization: The authors found that "plain" group-lasso actually outperformed a more complex "cycle-cost-weighted" lasso. The simpler approach removed more work and resulted in better speech quality.
  • Hardware-Software Co-design: By using a "table-and-tape" representation, the learned precisions and sparsity patterns were carried directly from the training script into the FPGA synthesis tools, ensuring bit-exact reproduction of the trained arithmetic on silicon.
Fig. 3. Share of scheduled MACs in surviving output channels by TCN block.

Key Results

The sparse model, SKIP-PLAIN, emerged as the clear winner:

  • Sparsity: It eliminated 47.3% of convolution MACs.
  • Latency: Frame processing time dropped from 1.86 ms to 1.03 ms (a 45% reduction).
  • Energy: Dynamic energy consumption fell by 60% (from 83 to 33 µJ/frame).
  • Quality: Despite the heavy compression, it achieved a PESQ score of 2.78, outperforming the 16-bit uncompressed reference (2.41) and dense quantized models.
  • Efficiency: The design used only 21 DSPs, compared to 899 used by prior state-of-the-art implementations on the same board.
Comparison of results across different model builds.

Limitations and Future Work

The primary limitation noted was training instability. Using Lasso regularization to prune channels can be volatile; the authors had to "freeze" the sparsity mask at epoch 50 to allow the remaining weights to stabilize.

Future work could explore more stable structured sparsity priors to avoid this manual freezing step. Additionally, while the frame processing time is very low, the system still operates on a 32ms window, which contributes to overall algorithmic latency—a factor always at odds with the "responsiveness" required for hearing aid applications.

Summary updated

Abstract

Hardware-aware training often optimizes FPGA resources, latency, or bit operations, but folding changes what compression saves: shared arithmetic decouples precision, weight storage, and scheduled work. We derive training objectives from the measured costs of a folded streaming speech-enhancement accelerator. Mixed-precision weights reduce on-chip storage; structured channel sparsity removes entire operations from the engine schedule. We deploy a bit-exact implementation on the Kria KV260. The best sparse model eliminates 47.3% of convolution MACs, cutting frame time by 45% (1.86 to 1.03 ms) and dynamic energy by 60% (83 to 33 $μ$J/frame) against a storage-priced model measured in the same session, with higher speech quality. Relative to the most efficient dense build measured separately, frame time falls by 42% and energy by 48%. Reducing weight width from 8 to 6 bits lowers dynamic power by 27% but changes frame time by less than 5%. Plain channel group-lasso removes more work than cycle-cost-weighted lasso. Thus, folded-accelerator training should price storage and removable work separately, rather than rely on a single bit-operation metric.
BibTeX
@misc{laroche2026realizing,
  author = {Clément Laroche},
  title = {{Realizing Low-Precision Storage and Sparse Execution in a Folded FPGA Speech Enhancer}},
  year = {2026},
  eprint = {2610.35144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35144},
}
Figure