Convolutional Neural Networks Based System for Urban Sound Tagging with Spatiotemporal Context

Jisheng Bai, Jianfeng Chen, Mou Wang, Xiao-Lei ZhangarXiv 2020 · arXiv

Abstract

Noise pollution significantly impacts urban quality of life, necessitating advanced monitoring technologies. Urban Sound Tagging (UST) aims to identify noise sources in recordings to map pollution distribution. While traditional methods rely solely on audio signals, this paper proposes a multimodal system using Convolutional Neural Networks (CNNs) integrated with spatiotemporal context. By combining diverse acoustic features with metadata (location and time) and employing data augmentation (Mixup), the proposed system achieves a macro-AUPRC of 0.775 on the DCASE2020 task5 dataset, a 22.6% improvement over the baseline.

Introduction

Urban Sound Tagging is critical for environmental protection but faces challenges due to the acoustic complexity of city environments and the neglect of contextual information. Urban sounds are often non-stationary and highly correlated with human activity patterns. For instance, certain sounds are more probable at specific times or locations. This paper addresses these issues by treating spatiotemporal metadata as a primary input alongside audio data, allowing the model to leverage the "where" and "when" of a recording to improve classification accuracy.

Dataset and the Challenge of Class Imbalance

The study utilizes the SONYC-UST-V2 dataset, consisting of recordings from New York City sensors categorized into eight coarse-level classes. A significant hurdle in UST is the extreme class imbalance, which can bias models toward frequently occurring sounds.

Distribution of recordings across coarse-level classes, highlighting the dominance of engine noise and the rarity of classes like 'Dog' or 'Saw'.

As shown in the distribution above, "Engine" sounds are ubiquitous, while categories like "Dog" and "Powered Saw" have far fewer samples. This imbalance often leads models to default to the majority class when acoustic signatures are ambiguous. To combat this, the authors employ Mixup data augmentation to create synthetic training samples by linearly interpolating between existing ones.

Proposed Multimodal System

The proposed architecture integrates audio features with spatiotemporal context through a multi-stage process that combines deep learning for audio with specialized encoding for metadata.

Detailed multimodal architecture showing the fusion of audio features (processed via CNNs) and spatiotemporal context (encoded via FC/LSTM).

1. Feature Extraction and Audio Processing

The system extracts multiple features, including Log-Mel, Log-linear, and Harmonic-Percussive Source Separation (HPSS) components. HPSS-h focuses on harmonic sounds like music, while HPSS-p captures percussive impact sounds. These are fed into a 9-layer Residual CNN (CNN9-Res). Residual blocks are employed to stabilize training and allow the network to learn complex mappings without gradient vanishing.

2. Spatiotemporal Context Encoding

Metadata such as Latitude, Longitude, Week, Day, and Hour are normalized and converted into an 85-dimensional vector. This vector is processed by a Spatiotemporal Context Encoder (using either Fully Connected layers or LSTMs) to generate a high-level representation of the recording's context.

3. Multimodal Fusion

The audio and context representations are concatenated before the final classification layer. This allows the model to use the spatiotemporal context as a prior, helping it distinguish between sounds that might be acoustically similar but contextually distinct.

Results and Discussion

The system achieved a macro-AUPRC of 0.775, significantly outperforming the DCASE2020 baseline (0.632) and other state-of-the-art models like GCNN (0.676).

  • Contextual Gains: The model learns that certain sounds are geographically and temporally "locked." This helps it ignore acoustic outliers that do not fit the expected pattern for a specific time or place.
  • Class-Specific Improvements: The "Dog" category saw a massive 1010.87% improvement in AUPRC, demonstrating the effectiveness of Mixup and context for rare classes. Conversely, "Human Voice" saw minimal gains (1.88%), as voices are ubiquitous across NYC, making context less discriminative.
  • Feature Specialization: HPSS-h was found to be superior for "Music" by isolating harmonic components, while Log-Mel performed best for "Engine" and "Alert" sounds.

Conclusion

Integrating spatiotemporal context with deep convolutional networks provides a robust framework for urban sound tagging. While the system significantly improves performance, challenges remain in distinguishing between overlapping mechanical sounds (e.g., saws vs. engines) and vocal-heavy music. Future work will focus on refining features to better separate these indistinct urban noise sources.

Summary updated

Abstract

Noise pollution has a significant impact on the quality of life of citizens and urban development, and its prevention and control has been widely valued. In order to better protect the environment, the analysis and monitoring technology of urban noise pollution needs to be further optimized. Urban Sound Tagging (UST) can help to map the distribution of noise pollution and therefore attracts more attentions. The goal of UST is to tag a recording, which is collected by the sensors from urban environment, and returns whether noise pollution is audible or not. In previous works, only sound signal is provided and used. However, the spatiotemporal context can offer more information. In this paper, we proposed convolutional neural networks (CNNs) based system for UST with spatiotemporal context. In our system, multiple features and spatiotemporal context are combined, and fed into a residual CNN to predict whether noise of pollution is present in a 10-second recording. To eliminate the imbalance problem of the dataset, a data augmentation method is applied during the training. Finally, a fusion strategy is adopted to further improve the performance of UST. We evaluated the proposed system on the DCASE2020 task5 dataset. The experimental results show that the proposed system significantly outperform the baseline system on the evaluation metrics.
BibTeX
@misc{bai2020convolutional,
  author = {Jisheng Bai and Jianfeng Chen and Mou Wang and Xiao-Lei Zhang},
  title = {{Convolutional Neural Networks Based System for Urban Sound Tagging with Spatiotemporal Context}},
  year = {2020},
  eprint = {2011.00175},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.00175},
}
Figure