Shuffling and Mixing Data Augmentation for Environmental Sound Classification

Tadanobu Inoue, Phongtharin Vinayavekhin, Shiqiang Wang, David Wood, Asim Munawar, Bong Jun Ko, Nancy Greco, Ryuki TachibanaDCASE 2019 · OpenReview

Abstract

Environmental sound classification (ESC) is often hindered by the scarcity and imbalance of labeled audio data. This paper introduces a novel data augmentation technique termed Shuffling and Mixing (SM). By randomizing the temporal order of audio segments and mixing segments from different recordings of the same class, the proposed method generates diverse training samples without introducing statistical bias or unnatural event densities. When combined with a specialized Convolutional Neural Network (CNN) architecture designed for spectral and temporal feature extraction, the SM technique significantly improves classification performance. The system achieved a macro-averaged F1 score of 89.95% on the DCASE 2018 Task 5 challenge, outperforming the baseline by over 5%.

Introduction

Deep learning models for environmental sound classification require substantial amounts of high-quality labeled data. However, datasets for sounds like "eating" or "vacuum cleaning" are typically small and imbalanced. Existing augmentation techniques, such as pitch shifting or "Mixup" (overlaying two sounds), have notable drawbacks. Standard Mixup often doubles the "event density," creating sounds that are busier than real-world recordings and potentially biasing the model. Furthermore, these methods often preserve the original temporal sequence, which may not be a defining characteristic for many environmental sound classes.

Methods

The researchers propose a two-step augmentation strategy called Shuffling and Mixing (SM):

  1. Segmentation: An audio clip is divided into smaller, equal-length segments (e.g., 2 seconds).
  2. Shuffling: The temporal order of these segments is randomized. This creates a new sequence that remains representative of the original sound class.
  3. Mixing: Segments are drawn from two different recordings of the same class. A mixing ratio determines the proportion of segments taken from each source.
The Shuffling and Mixing process based on shuffled sequence and sound IDs.

This approach allows for the generation of "busier" or "quieter" variations while maintaining the overall average density of the training set.

Complementing this augmentation is a specialized CNN architecture. It utilizes initial layers that convolve across the frequency axis to capture local spectral patterns, followed by layers that convolve across the time axis to capture temporal dependencies.

Key Insights: Temporal Independence

A core insight of this work is that environmental sounds are often composed of discrete, unordered events. For instance, in an "eating" class, the sequence of a plate clinking and a fork dropping does not change the classification.

Visualization of swapping sound event orders to generate new training samples.

By shuffling segments, the model learns to focus on the presence of these events rather than their specific order, leading to greater robustness.

Results

The method was evaluated on two major benchmarks:

  • DCASE 2018 Task 5: The system achieved a macro-averaged F1 score of 89.95%, a significant improvement over the 84.50% baseline. This performance was instrumental in winning the challenge.
  • Urban Sound 8K: The SM technique proved competitive with established methods like pitch shifting and particularly improved accuracy for classes like "Street Music" and "Engine Idling."

Ablation studies confirmed that the performance gains were a result of both the new CNN architecture and the SM augmentation working in tandem.

Conclusion

The Shuffling and Mixing technique provides a robust way to augment environmental sound data by leveraging the temporal independence of sound events. While highly effective for many classes, future work may explore its applicability to complex acoustic scenes with inherent temporal progressions and optimize segment lengths to avoid artifacts.

Summary updated

BibTeX
@inproceedings{inoue2019shuffling,
  author = {Tadanobu Inoue and Phongtharin Vinayavekhin and Shiqiang Wang and David Wood and Asim Munawar and Bong Jun Ko and Nancy Greco and Ryuki Tachibana},
  title = {{Shuffling and Mixing Data Augmentation for Environmental Sound Classification}},
  booktitle = {DCASE},
  year = {2019},
  url = {https://openreview.net/forum?id=jkjsIpA_nQ},
}
Figure