Sub-band Convolutional Neural Networks for Small-footprint Spoken Term Classification

Chieh-Chi Kao, Ming Sun, Yixin Gao, Shiv Vitaladevuni, Chao Wang

Abstract

This paper proposes a Sub-band Convolutional Neural Network (CNN) for spoken term classification. While CNNs are effective in acoustic applications, the spatial invariance property of 2D kernels—standard in computer vision—does not translate well to audio features where the meaning of a kernel varies along the frequency axis. We propose a sub-band CNN architecture that applies specialized kernels to different frequency sub-bands, improving computational efficiency. Experimental results on the Speech Commands dataset demonstrate that this approach reduces computation by 39.7% for commands and 49.3% for digits classification while maintaining accuracy, particularly benefiting small-footprint models.

Introduction: The Frequency Challenge in Audio CNNs

In computer vision, CNNs rely on spatial invariance: a feature detector for a "cat ear" should work regardless of where the ear appears in an image. However, in audio spectrograms, the vertical axis represents frequency (pitch). A sound pattern at a low frequency (like a vowel) is fundamentally different from one at a high frequency (like a fricative 's' sound). Standard CNNs ignore this, sweeping the same kernels across the entire frequency spectrum, which is both physically counter-intuitive and computationally inefficient for small-footprint devices like smart speakers.

Proposed Architecture: Overlapped Sub-band CNN

To address the mismatch between spatial invariance and acoustic features, we propose an Overlapped Sub-band CNN. The architecture involves three key innovations:

  1. Frequency Band Splitting: The input feature map (e.g., 40-dimensional MFCCs) is divided into $B$ overlapping sub-bands.
  2. Specialized Kernels: Instead of a single set of filters for the whole map, the first layer applies independent convolutional kernels to each sub-band. This allows the model to learn frequency-specific patterns more effectively.
  3. Channel-wise Concatenation: After the first convolution, feature maps from all sub-bands are concatenated along the channel axis rather than the feature axis. This allows subsequent layers to integrate information across the entire frequency range holistically.
Comparison of CNN architectures: Full-band (left), Sub-band with feature concatenation (middle), and the proposed Sub-band with channel concatenation (right).

By overlapping the bands, we prevent information loss at the boundaries. The choice of channel-wise concatenation is critical; it effectively increases the receptive field in the frequency dimension for the second layer, allowing it to "see" across multiple bands simultaneously.

Experimental Results

The architecture was evaluated using the Google Speech Commands dataset (v0.02) on two tasks: 12-way Commands classification and 10-way Digits classification.

The results highlight a significant advantage for the sub-band approach, especially in resource-constrained scenarios. As shown in the performance curves, the overlapped sub-band model consistently achieves higher accuracy for a given computational budget (measured in FLOPS) compared to the traditional full-band baseline.

Accuracy vs. computational cost (FLOPS) for Digits (left) and Commands (right) classification tasks.

Key Performance Metrics:

  • Efficiency Gains: To match the baseline accuracy, the sub-band architecture required 39.7% fewer FLOPS for commands and 49.3% fewer FLOPS for digits.
  • Small-Footprint Robustness: The performance gap is most pronounced at lower FLOPS counts. While standard CNN performance degrades sharply as parameters are reduced, the sub-band model maintains high accuracy.
  • Optimal Configuration: Empirical testing showed that using 3 sub-bands provided the optimal trade-off between model complexity and classification performance.

Conclusion

Sub-band CNNs offer a more physically grounded and computationally efficient way to process audio features. By breaking the assumption of frequency-axis spatial invariance, we can design models that are significantly leaner without sacrificing performance. This makes the architecture ideal for "always-on" keyword spotting and spoken term classification on edge devices with limited power and memory.

Summary updated

Abstract

This paper proposes a Sub-band Convolutional Neural Network for spoken term classification. Convolutional neural networks (CNNs) have proven to be very effective in acoustic applications such as spoken term classification, keyword spotting, speaker identification, acoustic event detection, etc. Unlike applications in computer vision, the spatial invariance property of 2D convolutional kernels does not fit acoustic applications well since the meaning of a specific 2D kernel varies a lot along the feature axis in an input feature map. We propose a sub-band CNN architecture to apply different convolutional kernels on each feature sub-band, which makes the overall computation more efficient. Experimental results show that the computational efficiency brought by sub-band CNN is more beneficial for small-footprint models. Compared to a baseline full band CNN for spoken term classification on a publicly available Speech Commands dataset, the proposed sub-band CNN architecture reduces the computation by 39.7% on commands classification, and 49.3% on digits classification with accuracy maintained.
BibTeX
@misc{kao2019sub,
  author = {Chieh-Chi Kao and Ming Sun and Yixin Gao and Shiv Vitaladevuni and Chao Wang},
  title = {{Sub-band Convolutional Neural Networks for Small-footprint Spoken Term Classification}},
  year = {2019},
  eprint = {1907.01448},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.01448},
}
Figure