TL;DR
This paper demonstrates that for "folded" FPGA accelerators—where a single compute engine is reused across multiple layers—standard hardware-aware training metrics like "bit-operations" are misleading. By specifically targeting weight storage and structured channel sparsity during training, the authors achieved a 60% reduction in dynamic energy and a 45% speedup on a Kria KV260, while actually improving speech quality compared to dense models.
What Problem Does This Paper Address?
Speech enhancement models for hearing aids and wearables need to be fast, private, and extremely energy-efficient. While FPGAs are ideal for this, a "spatial" implementation (where every layer of a neural network gets its own dedicated hardware) often consumes more resources than a small device can provide.
The solution is folding: reusing one physical compute engine to process the network's blocks in sequence. However, most "hardware-aware" training methods optimize for proxy metrics like the total number of bit-operations. In a folded architecture, these proxies fail. For example, if your hardware engine uses a fixed 8-bit multiplier, reducing a specific layer's weights to 4 bits saves storage space but doesn't necessarily speed up the computation. To truly optimize a folded engine, training must focus on two specific hardware costs: on-chip storage (how many bits are packed into memory) and scheduled cycles (how many operations can be skipped entirely).
Methods
The authors implement Conv-FSENet, a dilated temporal convolutional network (TCN), on an AMD Kria KV260 FPGA.
Key technical components include:
- The Folded Engine: Instead of nine separate TCN blocks, they built one engine that fetches block-specific weights and history at each iteration.
- HGQ-HLS4ML Flow: They used High Granularity Quantization (HGQ) to learn optimal bit-widths for every tensor and channel.
- Engine-Aware Training: They modified the training loss function to include terms for the actual "tape" size (weight storage) and the number of MAC (multiply-accumulate) cycles.
- Structured Sparsity: They applied group-lasso regularization to drive entire output channels to zero. In a folded engine, if a channel is exactly zero, the hardware controller can skip that entire loop in the schedule, directly translating sparsity into lower latency.
Datasets
The models were trained and evaluated using the 16 kHz VoiceBank-DEMAND dataset, a standard benchmark for speech enhancement. The evaluation used a speaker-disjoint test set of 824 utterances to ensure the model generalizes to new voices.
Key Insights and Ideas
- Sparsity is the Latency Killer: In a folded design, reducing bit-width saves power and storage but not time. Only structured sparsity—removing entire rows from the MAC schedule—meaningfully shortens the execution time.
- The "Aha" Moment on Regularization: The authors found that "plain" group-lasso actually outperformed a more complex "cycle-cost-weighted" lasso. The simpler approach removed more work and resulted in better speech quality.
- Hardware-Software Co-design: By using a "table-and-tape" representation, the learned precisions and sparsity patterns were carried directly from the training script into the FPGA synthesis tools, ensuring bit-exact reproduction of the trained arithmetic on silicon.
Key Results
The sparse model, SKIP-PLAIN, emerged as the clear winner:
- Sparsity: It eliminated 47.3% of convolution MACs.
- Latency: Frame processing time dropped from 1.86 ms to 1.03 ms (a 45% reduction).
- Energy: Dynamic energy consumption fell by 60% (from 83 to 33 µJ/frame).
- Quality: Despite the heavy compression, it achieved a PESQ score of 2.78, outperforming the 16-bit uncompressed reference (2.41) and dense quantized models.
- Efficiency: The design used only 21 DSPs, compared to 899 used by prior state-of-the-art implementations on the same board.
Limitations and Future Work
The primary limitation noted was training instability. Using Lasso regularization to prune channels can be volatile; the authors had to "freeze" the sparsity mask at epoch 50 to allow the remaining weights to stabilize.
Future work could explore more stable structured sparsity priors to avoid this manual freezing step. Additionally, while the frame processing time is very low, the system still operates on a 32ms window, which contributes to overall algorithmic latency—a factor always at odds with the "responsiveness" required for hearing aid applications.
Summary updated