Semi-Supervised Audio Classification with Consistency-Based Regularization

Kangkang Lu, Chuan-Sheng Foo, Kah Kuan Teh, Huy Dat Tran, Vijay Ramaseshan Chandrasekhar

Abstract

Deep learning models for audio classification typically require vast amounts of manually labeled data. This paper adapts the "Mean Teacher" semi-supervised learning (SSL) framework to the audio domain, introducing specific perturbations like environmental noise and sample mixing. The results demonstrate that consistency-based regularization significantly reduces the labeling burden, achieving near-state-of-the-art performance with only a fraction of the original training labels.

Introduction

Building high-performing audio classification models is often bottlenecked by the cost and time required for manual labeling. While Semi-Supervised Learning (SSL) has seen great success in computer vision by leveraging unlabeled data, its application to audio is less straightforward. Common image perturbations, such as rotation or flipping, do not have direct equivalents in the acoustic domain. This research investigates how consistency-based SSL can be effectively tailored for audio by identifying robust, domain-specific perturbations.

Methods: The Mean Teacher Framework

The authors utilize the Mean Teacher architecture, which consists of two neural networks: a Student and a Teacher. The Student model learns from labeled data via standard classification loss. The Teacher model's weights are an Exponential Moving Average (EMA) of the Student's weights, providing a stable target for consistency regularization.

Mean Teacher architecture with sample mixing (mixup) for audio classification.

The core mechanism involves feeding both models the same unlabeled audio clip with different random perturbations. The Student is penalized via a consistency loss if its prediction deviates from the Teacher's. Key perturbations introduced include:

  • Time and Frequency Shifts: Spectrogram translations horizontally or vertically.
  • Environmental Noise: Mixing in real-world sounds (e.g., traffic, household noises) rather than simple white noise.
  • Sample Mixing (Mixup): Creating hybrid samples by linearly combining two clips and requiring the model to produce consistent combined predictions.

Datasets and Experimental Setup

The framework was evaluated on two major benchmarks:

  1. Google Speech Commands: 30 spoken command classes (e.g., "Yes", "No", "Up").
  2. UrbanSound8K: 10 classes of urban environmental sounds (e.g., car horns, dogs barking).

Key Results and Discussion

The experiments highlight the efficiency of SSL in low-data regimes. On the Google Speech Commands dataset, the Mean Teacher model achieved 92.45% accuracy using only 600 labels (1% of the total), significantly outperforming the supervised baseline of 87.04%.

Comparison of Mean Teacher performance with different noise types on Google Speech Commands.

As shown in the table above, the type of noise used for perturbation is critical. While standard Gaussian noise provides minimal benefit, incorporating diverse urban noise (MT+urban noise) pushed the 600-label accuracy to 94.31%. This suggests that real-world noise helps the model better distinguish the primary signal from complex backgrounds.

Furthermore, the "25% Rule" observed in the study indicates that the Mean Teacher model with only a quarter of the labels can match the performance of a fully supervised model using 100% of the data. On the UrbanSound8K dataset, the SSL approach improved accuracy by nearly 9% over the supervised baseline in low-label scenarios.

Conclusion

Consistency-based regularization, when paired with domain-appropriate perturbations like environmental noise and mixup, is a powerful tool for audio classification. These methods allow models to develop robust decision boundaries and ignore irrelevant background signals. Future work may extend these techniques to more complex tasks such as sound event detection or automated speech recognition, potentially exploring non-linear methods for audio sample mixing.

Summary updated

Abstract

Consistency-based semi-supervised learning methods such as the Mean Teacher method are state-of-the-art on image datasets, but have yet to be applied to audio data. Such methods encourage model predictions to be consistent on perturbed input data. In this paper, we incorporate audio-specific perturbations into the Mean Teacher algorithm and demonstrate the effectiveness of the resulting method on audio classification tasks. Specifically, we perturb audio inputs by mixing in other environmental audio clips, and leverage other training examples as sources of noise. Experiments on the Google Speech Command Dataset and UrbanSound8K Dataset show that the method can achieve comparable performance to a purely supervised approach while using only a fraction of the labels.
BibTeX
@inproceedings{lu2019semi,
  author = {Kangkang Lu and Chuan-Sheng Foo and Kah Kuan Teh and Huy Dat Tran and Vijay Ramaseshan Chandrasekhar},
  title = {{Semi-Supervised Audio Classification with Consistency-Based Regularization}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1231},
}
Figure