Slow-Fast Auditory Streams For Audio Recognition

Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, Dima Damen

Abstract

We propose a two-stream convolutional network for audio recognition, that operates on time-frequency spectrogram inputs. Following similar success in visual recognition, we learn Slow-Fast auditory streams with separable convolutions and multi-level lateral connections. The Slow pathway has high channel capacity while the Fast pathway operates at a fine-grained temporal resolution. We showcase the importance of our two-stream proposal on two diverse datasets: VGG-Sound and EPIC-KITCHENS-100, and achieve state-of-the-art results on both.

Introduction

Recognizing activities from audio—such as distinguishing between "cutting an onion" and "cutting cheese"—presents significant challenges due to the varied nature of sounds, which can be momentary, repetitive, or obscured by noise. Traditional deep learning models often treat audio spectrograms as standard images, applying 2D convolutions with square filters. However, this approach ignores the fundamental physical differences between the temporal (horizontal) and spectral (vertical) axes of a spectrogram.

Drawing inspiration from neuroscience, which suggests the human brain utilizes a ventral stream for identification ("what") and a dorsal stream for localization and timing ("where/when"), we propose an architecture that mimics this dual-pathway processing for improved computational audio recognition.

The Slow-Fast Auditory Network

The core of our approach is the Slow-Fast Auditory Network, which processes audio through two parallel pathways designed with complementary characteristics.

Overview of the Slow-Fast Auditory Network architecture, featuring parallel Slow and Fast pathways with lateral connections.

Pathway Design

  1. The Slow Pathway: This stream operates at a low temporal sampling rate but maintains high channel capacity. It is optimized for capturing "spectral semantics"—the detailed frequency patterns essential for identifying specific sound sources, such as bird species.
  2. The Fast Pathway: In contrast, this stream operates at a high temporal resolution (4x that of the Slow stream) with reduced channel capacity. It is designed to capture fine-grained temporal structures, such as rhythmic percussive sounds.

Architectural Innovations

To respect the non-homogeneous nature of spectrograms, we utilize separable convolutions, replacing standard $k \times k$ filters with $1 \times k$ and $k \times 1$ filters to process frequency and time independently. Furthermore, multi-level lateral connections allow the Fast stream to inject high-frequency temporal information into the Slow stream at various depths, facilitating a more robust feature representation than simple late-stage fusion.

Experimental Results

We evaluated the model on two large-scale benchmarks: VGG-Sound (over 200,000 clips) and EPIC-KITCHENS-100 (egocentric kitchen activities). The Slow-Fast architecture achieved state-of-the-art performance on both, reaching 52.46% Top-1 accuracy on VGG-Sound and 15.44% Action Top-1 accuracy on EPIC-KITCHENS-100.

Complementary Stream Strengths

A key finding of our research is the specialized performance of each stream. Analysis shows that the Slow stream excels at identifying animals and environmental scenes characterized by distinct frequency signatures. Conversely, the Fast stream is superior for percussive, articulated, or rhythmic sounds.

Class-level analysis showing sound categories where the Slow or Fast stream performs better.

Ablation studies further confirmed that this architectural specialization is more effective than simply increasing model depth. For instance, our two-stream model outperformed a baseline with nearly double the parameters that used two identical "Slow" streams, highlighting the efficiency of the "Fast" pathway.

Conclusion

The Slow-Fast Auditory Network demonstrates that mimicking biological dual-pathway processing leads to significant gains in audio recognition tasks. By specializing pathways for spectral detail and temporal precision, we achieve superior performance and efficiency. Future work will explore dynamic parameter learning for striding and channel ratios, as well as the potential for a unified audio-visual Slow-Fast framework.

Reference: Kazakos, E., Nagrani, A., Zisserman, A., & Damen, D. (2021). Slow-Fast Auditory Streams For Audio Recognition. arXiv:2103.03516

Summary updated

Abstract

We propose a two-stream convolutional network for audio recognition, that operates on time-frequency spectrogram inputs. Following similar success in visual recognition, we learn Slow-Fast auditory streams with separable convolutions and multi-level lateral connections. The Slow pathway has high channel capacity while the Fast pathway operates at a fine-grained temporal resolution. We showcase the importance of our two-stream proposal on two diverse datasets: VGG-Sound and EPIC-KITCHENS-100, and achieve state-of-the-art results on both.
BibTeX
@inproceedings{kazakos2021slow,
  author = {Evangelos Kazakos and Arsha Nagrani and Andrew Zisserman and Dima Damen},
  title = {{Slow-Fast Auditory Streams For Audio Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413376},
}
Figure