"It os Okay to be Uncommon": Quantizing Sound Event Detection Networks on Hardware Accelerators with Uncommon Sub-Byte Support

Yushu Wu, Xiao Quan, Mohammad Rasool Izadi, Chuan-Che Jeff Huang

TL;DR

This paper introduces a novel approach to significantly reduce the memory, latency, and energy consumption of sound event detection (SED) models on low-power headphones. By leveraging new hardware accelerators that support "uncommon" sub-byte bit-widths (like 3-bit) and employing a smart neural architecture search, they achieve substantial efficiency gains without sacrificing accuracy, paving the way for more sophisticated audio AI on edge devices.

What Problem Does This Paper Address?

Modern noise-canceling headphones are becoming increasingly intelligent, aiming to understand our audio environments to dynamically adjust noise cancellation, optimize audio equalization, and even alert us to important sounds like doorbells. However, running these sophisticated deep learning models on resource-constrained headphones (limited energy and on-chip memory) is a major challenge. While quantization (reducing the precision of model weights) is a common solution, standard 8-bit quantization often isn't enough for more complex models. Binary networks (1-bit) offer extreme efficiency but drastically reduce accuracy. The authors aim to find a sweet spot by exploring "uncommon" sub-byte bit-widths (like 3, 5, 6, or 7 bits) and developing a method to find the optimal bit-width for each layer of a network, specifically for SED tasks.

Methods

The core of this work lies in two key innovations:

  1. Hardware Support for Uncommon Bit-widths: The authors identify a new class of hardware accelerators, specifically the NE16 on the GAP9 processor, that can efficiently handle weights quantized to uncommon bit-widths (e.g., 3 bits), not just the standard power-of-two bit-widths (1, 4, 8). This is crucial because without hardware support, these uncommon bit-widths would require padding and result in no actual efficiency gains.
  2. FracBit: Differentiable Neural Architecture Search (NAS): To find the optimal bit-width for each layer of a network, they employ a NAS technique called FracBit. Unlike previous NAS methods that struggled with convergence or had large search spaces, FracBit uses fractional bit-widths during training and back-propagation. This allows gradients to better guide the search towards the optimal bit-width for each layer, leading to more stable training and better results. They also incorporate a memory constraint into the loss function to guide the search towards memory-efficient models.

Datasets

The experiments were conducted using a compiled dataset derived from 7 different sources, resulting in 211.6K audio files (673.4 hours). This dataset was processed into log-Mel spectrograms. For training and evaluation, they used:

  • Generic SED: A subset of the dataset containing 56 low-level SED classes.
  • Few-Shot Learning (SED & Speaker Identification): The full dataset, including unique voices from 1263 speakers. They employed an episodic-batch training strategy with a 100-episode, 12-way, 5-shot setup for few-shot learning.

Key Insights and Ideas

  • "Uncommon" is the New "Optimal": The paper strongly suggests that moving beyond standard 8-bit quantization and exploring bit-widths like 3, 5, 6, and 7 bits, when supported by hardware, can unlock significant efficiency gains without compromising accuracy.
  • Hardware-Software Co-design is Key: The benefits of uncommon bit-widths are only realized when the hardware (like the NE16 accelerator) is designed to support them. This highlights the importance of co-designing hardware and software for edge AI.
  • Layer-wise Bit-width Optimization Matters: Not all layers in a neural network have the same sensitivity to quantization. FracBit's ability to find optimal, per-layer bit-widths is crucial for achieving the best trade-off between efficiency and accuracy.
  • SED Tasks Benefit from Sub-Byte Quantization: The study demonstrates that SED tasks, including both generic classification and more complex few-shot learning (which involves embeddings), can effectively leverage sub-byte quantization.

Key Results

The paper reports impressive results when evaluating quantized models on actual hardware (GAP9 with NE16):

  • Memory Reduction: Achieved 54-69% memory reduction compared to 8-bit models, with FracBit finding models that offered an additional ~20% memory reduction over fixed-bitwidth approaches.
  • Latency Reduction: Achieved 45-47% reduction in inference latency compared to 8-bit models.
  • Energy Reduction: Achieved 53-69% reduction in energy consumption compared to 8-bit models.
  • Accuracy Preservation: These significant efficiency gains were achieved while maintaining floating-point performance (i.e., no accuracy drop).
  • Uncommon Bit-widths Utilized: The searched models indeed leveraged uncommon bit-widths (e.g., 3, 6, 7 bits) in various layers, demonstrating the effectiveness of the FracBit search.

Limitations and Future Work

  • Hardware Dependency: The significant benefits are contingent on the availability of hardware accelerators that support uncommon sub-byte operations. The authors acknowledge that without such hardware, padding would negate the efficiency gains.
  • Generalizability to Other Tasks: While the paper focuses on SED, the authors suggest that their findings and methods could be applicable to other audio processing tasks or even different domains like image recognition, though further evaluation would be needed.
  • Search Time: While FracBit is more efficient than some prior NAS methods, the search process still adds to the overall development time. Future work could focus on further optimizing the NAS efficiency.

Summary updated

Abstract

If our noise-canceling headphones can understand our audio environments, they can then inform us of important sound events, tune equalization based on the types of content we listen to, and dynamically adjust noise cancellation parameters based on audio scenes to further reduce distraction. However, running multiple audio understanding models on headphones with a limited energy budget and on-chip memory remains a challenging task. In this work, we identify a new class of neural network accelerators (e.g., NE16 on GAP9) that allows network weights to be quantized to different common (e.g., 8 bits) and uncommon bit-widths (e.g., 3 bits). We then applied a differentiable neural architecture search to search over the optimal bit-widths of a network on two different sound event detection tasks with potentially different requirements on quantization and prediction granularity (i.e., classification vs. embeddings for few-shot learning). We further evaluated our quantized models on actual hardware, showing that we reduce memory usage, inference latency, and energy consumption by an average of 62%, 46%, and 61% respectively compared to 8-bit models while maintaining floating point performance. Our work sheds light on the benefits of such accelerators on sound event detection tasks when combined with an appropriate search method.
BibTeX
@inproceedings{wu2024it,
  author = {Yushu Wu and Xiao Quan and Mohammad Rasool Izadi and Chuan-Che Jeff Huang},
  title = {{"It os Okay to be Uncommon": Quantizing Sound Event Detection Networks on Hardware Accelerators with Uncommon Sub-Byte Support}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=JR4msghLBI},
}

Loading related content...

Figure