TL;DR
This paper presents a deep learning framework for autonomous vehicles to detect and locate emergency vehicle sirens and car horns in noisy urban environments. By treating audio spectrograms as images and using a multi-task learning approach with a U-Net architecture, the system achieves high accuracy in classifying these critical sounds and accurately estimating their direction, even in extremely challenging low-SNR conditions.
What Problem Does This Paper Address?
Autonomous vehicles, despite their advanced visual and sensor capabilities, are often "deaf" to crucial auditory cues in their environment. Specifically, detecting and localizing sounds like emergency vehicle sirens and car horns is vital for safe navigation, enabling proactive maneuvers like yielding or preparing for unexpected events. However, urban soundscapes are notoriously noisy and unpredictable, making traditional audio processing techniques struggle to reliably isolate and identify these important alerts. This paper aims to bridge this gap by developing a robust auditory perception system for vehicles.
Methods
The core of the proposed system lies in a novel application of image processing techniques to audio data. Instead of traditional signal processing, the authors convert audio signals into "gammatonegrams" – a time-frequency representation inspired by human auditory perception. These gammatonegrams are then treated as images and fed into a deep learning architecture.
The key innovation is a multi-task learning (MTL) scheme employing a U-Net architecture. This U-Net performs two tasks simultaneously:
- Semantic Segmentation: It acts as a sophisticated noise-filtering mechanism. By learning to segment the gammatonegram, it effectively separates the target sound (siren or horn) from the background traffic noise. This is a significant improvement over previous methods that relied on simpler clustering (like k-means) which struggled in low-SNR scenarios.
- Acoustic Event Classification: Concurrently, the network classifies the segmented sound into one of three categories: siren, horn, or other traffic noise.
For sound source localization, the denoised gammatonegrams from the stereo microphones are used. A separate Convolutional Neural Network (CNN) takes the cross-correlation of these clean gammatonegrams as input to regress the direction of arrival (DoA) of the sound on the horizontal plane. This direct mapping approach is more robust to interference than traditional methods that rely on estimating interaural time and level differences.
Datasets
The authors collected four hours of driving data in Oxford, UK, using Knowles omnidirectional boom microphones. To create challenging evaluation scenarios, they also synthetically corrupted this recorded traffic noise with target sounds (sirens and horns) at various Signal-to-Noise Ratios (SNRs). Additionally, they utilized data from the Urban Sound Dataset and freesound.org for training and testing, focusing on clean signals for ground truth. The final dataset comprised over 30,000 samples, each 0.5 seconds long, covering sirens, horns, and other traffic sounds.
Key Insights and Ideas
- Spectrograms as Images: Treating audio spectrograms as images opens up powerful computer vision techniques, like CNNs and semantic segmentation, for audio analysis. This allows leveraging context across time and frequency, which is crucial for robust noise handling.
- Multi-Task Learning for Synergy: Combining denoising (segmentation) and classification within a single MTL framework allows the tasks to inform each other. This synergy leads to more accurate segmentation and classification, especially in noisy conditions.
- U-Net for Robust Denoising: The U-Net architecture's ability to learn complex patterns and reconstruct features makes it highly effective at separating target sounds from challenging background noise, outperforming simpler methods like k-means, particularly at low SNRs.
- Direct DoA Regression: Learning a direct mapping from denoised audio features to sound source direction, rather than relying on intermediate acoustic cues (like ITD/ILD), makes the localization more resilient to noise and inaccuracies in the denoising step.
- Human Auditory Perception as a Guide: Using gammatone filterbanks, which mimic the human cochlea, provides a natural pre-filtering step that is well-suited for detecting sounds designed to be heard by humans.
Key Results
- Classification Accuracy: The system achieved an impressive average classification rate of 94%.
- Localization Accuracy:
- With 0.5s audio frames, the median absolute error for direction of arrival (DoA) estimation was 7.5°.
- With longer 2.5s frames, this error reduced significantly to 2.5°.
- Robustness to Noise: The system demonstrated excellent performance even in scenarios with remarkably high noise levels, achieving accurate localization at SNRs as low as -40dB. This is a significant improvement over previous methods that struggled below -15dB.
- Comparison to Baselines: The proposed MTL approach outperformed baseline methods (FS and Mono networks) in DoA estimation, showing improvements of up to 46% and 42% respectively, while only slightly impacting classification accuracy. It also significantly outperformed SVM and RF-based systems, especially at lower SNRs.
Limitations and Future Work
- Horn vs. Siren Accuracy: The paper notes that localization is more accurate for sirens than for horns, as sirens have more distinct and stable acoustic characteristics. Horns, being more tonal, present a greater challenge for the system.
- Real-time Performance: While not directly analyzed, the authors anticipate that the system's real-time performance will be adequate, operating at approximately 2 Hz (dictated by the 0.5s audio frame size).
- Future Work: The authors suggest investigating the detection of other audio events and the handling of simultaneous acoustic alarms of different natures. They also mention the potential to extend the localization to a full 360° range with additional microphones.
Summary updated