Introduction
In the field of audio AI, pretrained models have significantly advanced classification tasks, such as identifying environmental sounds. However, a "benefit gap" remains for restoration tasks like speech enhancement. While classification models often use "encoder-only" architectures, restoration requires generating high-quality audio outputs. Researchers from Sony and Kyoto University investigated whether the Audio Masked Autoencoder (MAE) architecture—originally designed for classification—could bridge this gap by leveraging its inherent ability to "fill in the blanks" of missing audio data.
Methodology: From Classification to Restoration
The core of this approach is the Vision Transformer Autoencoder (ViT-AE). In the standard Audio MAE framework, audio is converted into a mel-spectrogram and divided into patches. A large portion of these patches is masked, and the model is trained to reconstruct the original signal.
To adapt this for Speech Enhancement (SE), the authors proposed three architectural variations:
- Vanilla ViT-AE: Performs a direct mapping from a noisy mel-spectrogram to a clean one.
- Residual Connections: The model learns to predict a noise component to subtract (additive) or a Time-Frequency (TF) mask to filter the input (multiplicative).
- STFT-oriented ViT-AE (ViT-AE-iSTFT): This version uses Short-Time Fourier Transform (STFT) magnitudes instead of mel-spectrograms, allowing for higher fidelity by incorporating phase information during the inverse STFT process.
A dual-scale loss function was also implemented to balance errors across logarithmic and linear scales, ensuring the model captures both coarse and fine audio details.
Datasets and Pretraining
The models were pretrained on two distinct datasets: AudioSet, containing 2 million diverse YouTube clips, and LibriTTS, a clean speech dataset. Evaluation was conducted on the Speech Command V2 (SPCv2) for classification and the Valentini dataset for speech enhancement. Additionally, the DAPS dataset was used for zero-shot testing to evaluate generalization to unseen environments.
Key Results and Insights
The study revealed that pretraining on the "noisy" AudioSet was remarkably effective, often matching or exceeding the performance of models pretrained on clean speech. This suggests that the model learns robust audio structures even from imperfect data.
In classification, the model achieved a state-of-the-art 97.8% accuracy on SPCv2. For restoration, the benefits of pretraining were clear across all metrics.
As shown in the results table, the multiplicative and additive residual models pretrained on AudioSet or LibriTTS significantly outperformed the "from-scratch" baselines. Specifically, the STFT-oriented variation showed superior signal fidelity (PESQ), while mel-to-mel models excelled in perceptual quality metrics like NISQA. The similarity between the MAE pretraining task and the restoration task allows for excellent generalization to out-of-domain distortions, as evidenced by strong zero-shot performance on the DAPS dataset.
Conclusion
By extending Audio MAEs to restoration, this research demonstrates the potential for a "universal" audio model capable of both understanding and fixing sound. Future work aims to bridge the domain mismatch between mel-pretraining and STFT-finetuning and to apply these techniques to broader tasks like music restoration and declipping.
Summary updated