A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging

Yun Wang (Maigo), Juncheng Li, Yun Wang, Joseph Szurley, Florian Metze, Samarjit Das

Introduction

Environmental audio tagging—the task of identifying sounds such as sirens, car horns, or tires screeching—is a critical component for smart city and surveillance applications. However, the field faces a significant data bottleneck. High-quality "strong labels" (precise timestamps) are expensive to produce. While "weak labels" (clip-level tags) are more accessible, they present a challenge for models that must discern which specific frames contain the relevant acoustic features.

Furthermore, existing multimodal approaches that leverage video context are often computationally prohibitive. Researchers from Carnegie Mellon and Bosch have addressed these challenges by developing a "light-weight" multimodal framework. Their system improves tagging accuracy by 5.3% in $F_1$ score while maintaining a training time of under six hours on a single GPU.

Methodology

The proposed framework utilizes a dual-branch architecture that fuses audio and visual information at the prediction level.

Audio Branch: CRNN with Multiple Instance Learning

The audio processing pipeline employs a Convolutional Recurrent Neural Network (CRNN) to analyze Mel spectral features. To address the lack of precise timestamps in weakly labeled data, the authors implement Multiple Instance Learning (MIL). In this paradigm, a 10-second audio clip is treated as a "bag" of frames. The model generates frame-level predictions, but the loss is calculated based on the maximum probability across the entire clip (max pooling), allowing the network to autonomously identify the most relevant acoustic segments.

Video Branch: Semantic Knowledge Mapping

To avoid the high cost of training a video-specific network, the system uses a "knowledge mapping" shortcut:

  1. Key Frame Extraction: A sparse coding algorithm selects the four most representative frames from a clip. The authors note that sparse coding is more robust for selecting keyframes than standard clustering methods like K-Means because it better handles outliers in the video stream.
  2. Object Recognition: These frames are processed by a pretrained Inception V4 model to identify visual objects.
  3. Semantic Mapping: Detected objects (e.g., "ambulance") are mapped to audio tags (e.g., "siren") using GloVe word embeddings. This allows the system to infer likely sounds based on visual context without explicit audio-visual training.
The multimodal architecture showing the fusion of the CRNN audio branch and the pretrained video branch.

Experimental Setup

The framework was evaluated using the DCASE 2017 Challenge (Task 4) dataset, which consists of approximately 51,000 YouTube excerpts (142 hours) from the Google Audio Set. The dataset includes 17 sound event types primarily focused on vehicles and warning signals. All training data utilized "weak" labels.

Results and Discussion

The multimodal system achieved a micro-average $F_1$ score of 55.9% on the evaluation set, a significant improvement over the 50.6% achieved by the audio-only baseline.

Visuals as a "Sanity Check"

A key finding was the complementary nature of the two modalities. Some sound classes are acoustically ambiguous but visually distinct. For instance, the "car passing by" class was frequently missed by the audio-only model but was easily identified by the video branch, leading to a substantial performance jump for that category.

Per-class performance comparison between the audio-only baseline and the multimodal system.

Efficiency and Speed

By leveraging off-the-shelf vision (Inception) and language (GloVe) models, the researchers minimized computational overhead. The video branch requires less than 30 minutes of processing, while the audio branch trains in roughly 6 hours on a single GTX 1080 Ti. This efficiency makes the framework highly accessible for researchers with limited compute resources.

Conclusion and Future Work

While the framework demonstrates clear benefits, it is not without limitations. The authors noted occasional "audio-visual mismatches," where visual cues (e.g., people talking about a bike) do not correspond to actual sounds. Future work may involve learning the fusion weights automatically rather than using manual tuning, and exploring cross-modal attention mechanisms to more precisely link visual movements to acoustic events.

Summary updated

Abstract

The lack of strong labels has severely limited the state-of-the-art fully supervised audio tagging systems to be scaled to larger dataset. Meanwhile, audio-visual learning models based on unlabeled videos have been successfully applied to audio tagging, but they are inevitably resource hungry and require a long time to train. In this work, we propose a light-weight, multimodal framework for environmental audio tagging. The audio branch of the framework is a convolutional and recurrent neural network (CRNN) based on multiple instance learning (MIL). It is trained with the audio tracks of a large collection of weakly labeled YouTube video excerpts; the video branch uses pretrained state-of-the-art image recognition networks and word embeddings to extract information from the video track and to map visual objects to sound events. Experiments on the audio tagging task of the DCASE 2017 challenge show that the incorporation of video information improves a strong baseline audio tagging system by 5.3% in terms of F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> score. The entire system can be trained within 6 hours on a single GPU, and can be easily carried over to other audio tasks such as speech sentimental analysis.
BibTeX
@inproceedings{maigo2017light,
  author = {Yun Wang (Maigo) and Juncheng Li and Yun Wang and Joseph Szurley and Florian Metze and Samarjit Das},
  title = {{A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  doi = {10.1109/icassp.2018.8462479},
}
Figure