Abstract
Sound Event Detection (SED) requires models to identify both the class and temporal boundaries of acoustic events. This paper explores the efficacy of applying data augmentations to the internal "latent" layers of a neural network compared to traditional input-level perturbations. By systematically evaluating various augmentation types—including random, adversarial, and Mixup—within a decoupled consistency-learning framework, the authors demonstrate that latent augmentations significantly enhance labeling accuracy. Furthermore, they show that a simplified training regime using standard $L_2$ regularization can match or exceed the performance of the more complex Mean Teacher framework while reducing computational overhead.
Introduction
Sound Event Detection (SED) is a critical task in audio processing, with applications ranging from security monitoring to healthcare. However, the field is hampered by the high cost of manual annotation and the inherent complexity of identifying overlapping sounds in noisy environments. To address these challenges, researchers often employ consistency-based learning, where a model is trained to produce stable predictions under small input perturbations.
While input-level augmentation is standard, this study investigates whether perturbing the internal representations (latent layers) of a model provides better regularization. The authors also address the "black box" nature of these models by exploring adversarial perturbations designed to challenge the model's decision boundaries.
Methodology
The researchers utilized a Convolutional Recurrent Neural Network (CRNN)—the state-of-the-art architecture for SED tasks. As illustrated in Figure 1, the architecture allows for the injection of noise at multiple stages: the input Mel spectrogram, the output of the CNN blocks, and the output of the RNN blocks.
The study compared several augmentation strategies:
- Location: Input vs. Latent levels.
- Perturbation Type: Random Gaussian Noise, Adversarial Noise (gradient-based), Virtual Adversarial Training (VAT), and Mixup.
- Regularization Framework: The standard Mean Teacher (MT) framework was compared against a simplified version that replaces the temporal weight averaging of MT with standard $L_2$ weight regularization. This decoupling allows "consistency" and "regularization" to be tuned independently.
The experiments were conducted on the Domestic Environment Sound Event Detection (DESED) dataset (DCASE 2022 Task 4), which includes a mix of weakly labeled, unlabeled, and synthetic strongly labeled audio.
Results and Discussion
The evaluation focused on two primary metrics: macro F1 (measuring temporal localization) and the Polyphonic Sound Detection Score (PSDS, measuring labeling accuracy).
As shown in Figure 2, several key insights emerged:
- Superiority of Latent Augmentation: Applying random noise to the latent layers (specifically after the CNN) achieved the highest labeling accuracy with a PSDS of 0.565, outperforming the input-level baseline (0.552).
- Benefits of Decoupling: Separating weight regularization from consistency costs allowed for better optimization. The best temporal accuracy (macro F1 of 0.397) was achieved using random input noise with separate $L_2$ regularization.
- Efficiency of Simplified Training: The simplified $L_2$ framework (Row 3 in Figure 2) not only outperformed the baseline but also halved the memory requirements by eliminating the need for a separate "teacher" model.
- The Adversarial Paradox: Adversarial augmentations did not yield the expected performance gains. The authors hypothesize that purely mathematical adversarial noise may create "unrealistic" samples that do not reflect real-world acoustic variations, suggesting a need for "audio-aware" adversarial attacks.
Conclusion
This research highlights the potential of latent-level data augmentation for improving Sound Event Detection. By moving beyond simple input perturbations and decoupling training objectives, models can achieve higher robustness and accuracy with lower computational costs. Future work will investigate more realistic, audio-specific adversarial perturbations to further bridge the gap between synthetic training and real-world performance.
Summary updated