Abstract
This paper proposes a Sub-band Convolutional Neural Network (CNN) for spoken term classification. While CNNs are effective in acoustic applications, the spatial invariance property of 2D kernels—standard in computer vision—does not translate well to audio features where the meaning of a kernel varies along the frequency axis. We propose a sub-band CNN architecture that applies specialized kernels to different frequency sub-bands, improving computational efficiency. Experimental results on the Speech Commands dataset demonstrate that this approach reduces computation by 39.7% for commands and 49.3% for digits classification while maintaining accuracy, particularly benefiting small-footprint models.
Introduction: The Frequency Challenge in Audio CNNs
In computer vision, CNNs rely on spatial invariance: a feature detector for a "cat ear" should work regardless of where the ear appears in an image. However, in audio spectrograms, the vertical axis represents frequency (pitch). A sound pattern at a low frequency (like a vowel) is fundamentally different from one at a high frequency (like a fricative 's' sound). Standard CNNs ignore this, sweeping the same kernels across the entire frequency spectrum, which is both physically counter-intuitive and computationally inefficient for small-footprint devices like smart speakers.
Proposed Architecture: Overlapped Sub-band CNN
To address the mismatch between spatial invariance and acoustic features, we propose an Overlapped Sub-band CNN. The architecture involves three key innovations:
- Frequency Band Splitting: The input feature map (e.g., 40-dimensional MFCCs) is divided into $B$ overlapping sub-bands.
- Specialized Kernels: Instead of a single set of filters for the whole map, the first layer applies independent convolutional kernels to each sub-band. This allows the model to learn frequency-specific patterns more effectively.
- Channel-wise Concatenation: After the first convolution, feature maps from all sub-bands are concatenated along the channel axis rather than the feature axis. This allows subsequent layers to integrate information across the entire frequency range holistically.
By overlapping the bands, we prevent information loss at the boundaries. The choice of channel-wise concatenation is critical; it effectively increases the receptive field in the frequency dimension for the second layer, allowing it to "see" across multiple bands simultaneously.
Experimental Results
The architecture was evaluated using the Google Speech Commands dataset (v0.02) on two tasks: 12-way Commands classification and 10-way Digits classification.
The results highlight a significant advantage for the sub-band approach, especially in resource-constrained scenarios. As shown in the performance curves, the overlapped sub-band model consistently achieves higher accuracy for a given computational budget (measured in FLOPS) compared to the traditional full-band baseline.
Key Performance Metrics:
- Efficiency Gains: To match the baseline accuracy, the sub-band architecture required 39.7% fewer FLOPS for commands and 49.3% fewer FLOPS for digits.
- Small-Footprint Robustness: The performance gap is most pronounced at lower FLOPS counts. While standard CNN performance degrades sharply as parameters are reduced, the sub-band model maintains high accuracy.
- Optimal Configuration: Empirical testing showed that using 3 sub-bands provided the optimal trade-off between model complexity and classification performance.
Conclusion
Sub-band CNNs offer a more physically grounded and computationally efficient way to process audio features. By breaking the assumption of frequency-axis spatial invariance, we can design models that are significantly leaner without sacrificing performance. This makes the architecture ideal for "always-on" keyword spotting and spoken term classification on edge devices with limited power and memory.
Summary updated