Abstract
Noise pollution significantly impacts urban quality of life, necessitating advanced monitoring technologies. Urban Sound Tagging (UST) aims to identify noise sources in recordings to map pollution distribution. While traditional methods rely solely on audio signals, this paper proposes a multimodal system using Convolutional Neural Networks (CNNs) integrated with spatiotemporal context. By combining diverse acoustic features with metadata (location and time) and employing data augmentation (Mixup), the proposed system achieves a macro-AUPRC of 0.775 on the DCASE2020 task5 dataset, a 22.6% improvement over the baseline.
Introduction
Urban Sound Tagging is critical for environmental protection but faces challenges due to the acoustic complexity of city environments and the neglect of contextual information. Urban sounds are often non-stationary and highly correlated with human activity patterns. For instance, certain sounds are more probable at specific times or locations. This paper addresses these issues by treating spatiotemporal metadata as a primary input alongside audio data, allowing the model to leverage the "where" and "when" of a recording to improve classification accuracy.
Dataset and the Challenge of Class Imbalance
The study utilizes the SONYC-UST-V2 dataset, consisting of recordings from New York City sensors categorized into eight coarse-level classes. A significant hurdle in UST is the extreme class imbalance, which can bias models toward frequently occurring sounds.
As shown in the distribution above, "Engine" sounds are ubiquitous, while categories like "Dog" and "Powered Saw" have far fewer samples. This imbalance often leads models to default to the majority class when acoustic signatures are ambiguous. To combat this, the authors employ Mixup data augmentation to create synthetic training samples by linearly interpolating between existing ones.
Proposed Multimodal System
The proposed architecture integrates audio features with spatiotemporal context through a multi-stage process that combines deep learning for audio with specialized encoding for metadata.
1. Feature Extraction and Audio Processing
The system extracts multiple features, including Log-Mel, Log-linear, and Harmonic-Percussive Source Separation (HPSS) components. HPSS-h focuses on harmonic sounds like music, while HPSS-p captures percussive impact sounds. These are fed into a 9-layer Residual CNN (CNN9-Res). Residual blocks are employed to stabilize training and allow the network to learn complex mappings without gradient vanishing.
2. Spatiotemporal Context Encoding
Metadata such as Latitude, Longitude, Week, Day, and Hour are normalized and converted into an 85-dimensional vector. This vector is processed by a Spatiotemporal Context Encoder (using either Fully Connected layers or LSTMs) to generate a high-level representation of the recording's context.
3. Multimodal Fusion
The audio and context representations are concatenated before the final classification layer. This allows the model to use the spatiotemporal context as a prior, helping it distinguish between sounds that might be acoustically similar but contextually distinct.
Results and Discussion
The system achieved a macro-AUPRC of 0.775, significantly outperforming the DCASE2020 baseline (0.632) and other state-of-the-art models like GCNN (0.676).
- Contextual Gains: The model learns that certain sounds are geographically and temporally "locked." This helps it ignore acoustic outliers that do not fit the expected pattern for a specific time or place.
- Class-Specific Improvements: The "Dog" category saw a massive 1010.87% improvement in AUPRC, demonstrating the effectiveness of Mixup and context for rare classes. Conversely, "Human Voice" saw minimal gains (1.88%), as voices are ubiquitous across NYC, making context less discriminative.
- Feature Specialization: HPSS-h was found to be superior for "Music" by isolating harmonic components, while Log-Mel performed best for "Engine" and "Alert" sounds.
Conclusion
Integrating spatiotemporal context with deep convolutional networks provides a robust framework for urban sound tagging. While the system significantly improves performance, challenges remain in distinguishing between overlapping mechanical sounds (e.g., saws vs. engines) and vocal-heavy music. Future work will focus on refining features to better separate these indistinct urban noise sources.
Summary updated