A Deep Residual Network for Large-Scale Acoustic Scene Analysis

Logan Ford, Hao Tang, François Grondin, James R. GlassInterspeech 2019 · DOI

Researchers from MIT’s CSAIL, presented at Interspeech 2019, have demonstrated a significant leap in large-scale audio classification. By moving away from "frozen" pre-computed features and embracing end-to-end deep learning, their model achieves state-of-the-art results on the AudioSet benchmark while demonstrating a unique ability to localize sounds in time.

The Challenge of Large-Scale Audio

In the field of machine hearing, AudioSet is the primary benchmark, containing over 2 million 10-second YouTube clips. Due to its massive scale, most researchers rely on 128-dimensional "embeddings" (bottleneck features) provided by Google. While efficient, this approach creates a "black box" where the underlying model cannot be fine-tuned, and it remains unclear if performance gains stem from new algorithms or Google's massive external pre-training.

This research addresses two core questions: Can a model trained from scratch on raw audio outperform the state-of-the-art? And can such a model learn to identify when a sound occurs using only "weak" labels (i.e., knowing what sound is in a clip, but not its exact timestamps)?

Methodology: ResNets and Channel Pooling

The authors utilized the ResNet (Residual Network) architecture, well-known in computer vision for its ability to train very deep networks via skip connections. To adapt this for audio, they implemented several key technical innovations:

  • End-to-End Training: The model bypasses pre-computed features, training directly on log-Mel spectrograms.
  • Channel Pooling: Standard models typically use Global Pooling, which averages out all time and frequency data at the final layer. The authors introduced "Channel Pooling," which averages across frequency channels but preserves the time dimension. This preservation is crucial for temporal localization.
  • Attention Mechanism: To handle the fact that a specific sound (like a dog bark) might only occupy a fraction of a 10-second clip, an attention module was added. This allows the network to "focus" on relevant segments before final classification.
Comparison between (a) Global pooling, which collapses temporal data, and (b) the proposed Channel pooling, which preserves the time dimension.

Key Results and Insights

The best-performing model, a ResNet-50 with Attention, achieved a mean Average Precision (mAP) of 0.392 and an Area Under Curve (AUC) of 0.971. These results surpassed previous benchmarks that relied on significantly more external pre-training data.

Weak to Strong Learning

One of the most significant findings was the model's ability to perform "weakly-supervised" localization. Although the model was only trained on clip-level labels (e.g., "this clip contains a cat"), the attention mechanism naturally learned to spike during the specific moments the sound occurred.

On the DCASE 2018 (Task 4) dataset, which provides "strong labels" (exact timestamps) for verification, the model significantly outperformed baselines for short-duration events. For instance, it achieved a 40.0% F1 score for "Cat" sounds compared to a 0.0% baseline, with a temporal resolution of approximately 300ms.

Limitations and Future Directions

While highly effective for impulsive or short sounds, the model faced challenges with long, continuous sounds like "vacuuming" or "frying." In these cases, the attention mechanism tended to trigger only on the most distinctive acoustic portions rather than the entire duration.

Future work may involve integrating Recurrent Neural Networks (RNNs) or LSTMs to better capture the long-term temporal structure of sustained acoustic scenes, potentially refining the model's understanding of complex, multi-layered audio environments.

Summary updated

BibTeX
@inproceedings{ford2019deep,
  author = {Logan Ford and Hao Tang and François Grondin and James R. Glass},
  title = {{A Deep Residual Network for Large-Scale Acoustic Scene Analysis}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2731},
}
Figure