Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training

Dading Chong, Helin Wang, Peilin Zhou, Qingcheng Zeng

Abstract

Transformer-based models attain excellent results and generalize well when trained on sufficient amounts of data. However, constrained by the limited data available in the audio domain, most transformer-based models for audio tasks are finetuned from pre-trained models in other domains (e.g. image), which has a notable gap with the audio domain. Other methods explore the self-supervised learning approaches directly in the audio domain but currently do not perform well in the downstream tasks. In this paper, we present a novel self-supervised learning method for transformer-based audio models, called masked spectrogram prediction (MaskSpec), to learn powerful audio representations from unlabeled audio data (AudioSet used in this paper). Our method masks random patches of the input spectrogram and reconstructs the masked regions with an encoder-decoder architecture. Without using extra model weights or supervision, experimental results on multiple downstream datasets demonstrate MaskSpec achieves a significant performance gain against the supervised methods and outperforms the previous pre-trained models. In particular, our best model reaches the performance of 0.471 (mAP) on AudioSet, 0.854 (mAP) on OpenMIC2018, 0.982 (accuracy) on ESC-50, 0.976 (accuracy) on SCV2, and 0.823 (accuracy) on DCASE2019 Task1A respectively.

Introduction

Transformer-based models are the current gold standard for many AI tasks, but they are notoriously data-hungry. In the audio domain, labeled data is scarce compared to the billions of images available in datasets like ImageNet. To compensate, researchers usually employ cross-domain transfer (initializing audio models with weights from image models) or self-supervised learning (SSL) on unlabeled audio. However, cross-domain transfer suffers from a fundamental "domain gap" between spectrograms and natural images, while previous audio SSL methods often lagged behind models pre-trained on ImageNet.

The authors of this paper introduce MaskSpec (Masked Spectrogram Prediction) to prove that a pure audio SSL approach can be just as powerful as cross-domain transfer if designed correctly.

The MaskSpec Framework

MaskSpec is inspired by the Masked Autoencoder (MAE) architecture. The process involves dividing an input log-Mel spectrogram into $16 \times 16$ patches and randomly masking a high percentage (75%) of them.

The MaskSpec architecture: random patches of the input spectrogram are masked, and an asymmetric encoder-decoder reconstructs the missing regions.

The architecture is asymmetric:

  • Encoder: A standard Transformer that only processes the visible (unmasked) patches, making training highly efficient.
  • Decoder: A lightweight Transformer that receives the encoder's output and "mask tokens" to reconstruct the original spectrogram.
  • Loss: The model is trained using Mean Squared Error (MSE) between the predicted and original pixels of the masked patches.

By forcing the model to predict missing sounds based on surrounding context, it develops a deep understanding of complex time-frequency structures.

Experimental Results

MaskSpec was pre-trained on the full unlabeled AudioSet (~2 million clips) and evaluated on several downstream tasks, including AudioSet-20K, ESC-50, DCASE2019 Task1A, OpenMIC2018, and Speech Commands V2 (SCV2).

Performance comparison of MaskSpec against state-of-the-art models across multiple downstream audio tasks.

Key findings include:

  • Competitive Performance: MaskSpec achieved a mAP of 0.471 on AudioSet, matching the state-of-the-art PaSST model which relies on ImageNet pre-training.
  • Generalization: On the ESC-50 dataset, MaskSpec reached 98.2% accuracy, setting a new benchmark for environmental sound classification.
  • Efficiency: Even the "Tiny" version (5.6M parameters) outperformed much larger CNN-based models, demonstrating the effectiveness of the SSL objective for smaller architectures.

Discussion and Key Insights

A critical finding was the Optimal Masking Ratio. The authors discovered that a 75% mask ratio is the "sweet spot." Lower ratios make the task too easy, while higher ratios (e.g., 95%) make reconstruction impossible. This high ratio confirms that audio spectrograms contain significant redundancy, allowing the model to learn rich contextual representations from sparse data.

While MaskSpec performs exceptionally well on general audio, its performance on the speech-heavy SCV2 dataset was slightly lower than SSAST, which utilizes extra speech-specific data (Librispeech). This suggests that while general audio pre-training is powerful, domain-specific data may still offer benefits for specialized tasks like speech recognition. Additionally, while the encoder is efficient, pre-training on the full AudioSet remains computationally intensive, requiring significant GPU resources.

Conclusion

MaskSpec demonstrates that self-supervised pre-training directly on audio data can effectively close the domain gap previously filled by ImageNet weights. By leveraging an asymmetric encoder-decoder and high-ratio masking, MaskSpec provides a powerful and efficient framework for learning audio representations that generalize across a wide variety of downstream tasks.

Summary updated

Abstract

Transformer-based models attain excellent results and generalize well when trained on sufficient amounts of data. However, constrained by the limited data available in the audio domain, most transformer-based models for audio tasks are finetuned from pre-trained models in other domains (e.g. image), which has a notable gap with the audio domain. Other methods explore the self-supervised learning approaches directly in the audio domain but currently do not perform well in the downstream tasks. In this paper, we present a novel self-supervised learning method for transformer-based audio models, called masked spectrogram prediction (MaskSpec), to learn powerful audio representations from unlabeled audio data (AudioSet used in this paper). Our method masks random patches of the input spectrogram and reconstructs the masked regions with an encoder-decoder architecture. Without using extra model weights or supervision, experimental results on multiple downstream datasets demonstrate MaskSpec achieves a significant performance gain against the supervised methods and outperforms the previous pre-trained models. In particular, our best model reaches the performance of 0.471 (mAP) on AudioSet, 0.854 (mAP) on OpenMIC2018, 0.982 (accuracy) on ESC-50, 0.976 (accuracy) on SCV2, and 0.823 (accuracy) on DCASE2019 Task1A respectively.
BibTeX
@inproceedings{chong2023masked,
  author = {Dading Chong and Helin Wang and Peilin Zhou and Qingcheng Zeng},
  title = {{Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=8BeaaTZ-SBe},
}
Figure