Robustdistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness

Heitor R. Guimarães, Arthur Pimentel, Anderson R. Avila, Mehdi Rezagholizadeh, Boxing Chen, Tiago H. Falk

Introduction

Self-supervised learning (SSL) has fundamentally changed the landscape of speech processing. Models such as HuBERT and wav2vec 2.0 learn universal representations from vast amounts of unlabeled audio, providing a powerful foundation for downstream tasks like speech recognition, emotion detection, and keyword spotting. However, these models face two significant barriers to real-world deployment: their massive size (often exceeding 95 million parameters) and their extreme sensitivity to environmental noise and reverberation.

While previous research has addressed these issues in isolation—using distillation for compression or data augmentation for robustness—RobustDistiller is the first framework to tackle both simultaneously. By combining layer-wise knowledge distillation with a novel multi-task denoising objective, RobustDistiller produces "student" models that are 75% smaller than their "teachers" yet significantly more resilient in noisy, real-world environments.

The Challenge: Size and Fragility

Universal speech representations are typically trained on clean, high-quality audio datasets like LibriSpeech. While this allows them to learn rich linguistic features, it leaves them fragile when faced with "distribution shifts"—the inevitable background noise, office chatter, or room echoes found in everyday life.

Furthermore, the computational requirements of these models make them unsuitable for "edge" devices such as smartphones or smart speakers. Existing compression techniques, like DistilHuBERT, successfully reduce model size but often inherit or even amplify the teacher model's sensitivity to noise. RobustDistiller aims to bridge this gap, creating a model that is both lightweight and "tough."

The RobustDistiller Framework

RobustDistiller introduces a training recipe that can be applied to any universal speech representation. The framework builds on standard knowledge distillation—where a small student model learns to mimic the internal states of a large teacher—but adds two critical innovations:

1. Online Contamination (Data Augmentation)

During the training process, the student model is fed audio that has been intentionally corrupted with additive noise (e.g., street sounds, music) and room impulse responses (reverberation). Crucially, while the student receives the "dirty" audio, it is still tasked with predicting the clean representations generated by the teacher model. This forces the student to learn how to ignore environmental artifacts and focus on the underlying speech signal.

2. Multi-Task Denoising Learning

To further enhance robustness, the authors added an "enhancement head" to the student model. This component, consisting of a BiLSTM and transposed convolutions, attempts to reconstruct the original clean waveform directly from the student's internal representations. This multi-task approach acts as a powerful regularizer; the student's encoder must preserve enough information to rebuild the clean signal, effectively "disentangling" speech features from noise at the representation level.

Experimental Setup and Results

The models were pre-trained on the 960-hour LibriSpeech corpus and evaluated using the SUPERB benchmark across three tasks: Keyword Spotting (KS), Intent Classification (IC), and Emotion Recognition (ER). To test real-world performance, the authors utilized noise from DCASE2020, DNS4, and simulated reverberation from OpenSLR26.

Key Findings:

  • Massive Compression: The resulting student models contain approximately 24 million parameters, making them roughly one-quarter the size of the 95-million-parameter teacher models.
  • The "Student Beats Teacher" Phenomenon: In noisy scenarios, the RobustDistiller student frequently outperformed its much larger teacher. For example, Robust DistilWavLM achieved an overall score of 73.69% across all conditions, surpassing its teacher, WavLM Base+ (72.02%), and significantly outperforming the standard distilled version (59.15%).
  • Internal vs. External Enhancement: The research demonstrates that internal feature-level enhancement is superior to traditional signal-level pre-processing. While cleaning audio before it reaches a model can introduce artificial distortions that hurt performance, RobustDistiller's internal denoising preserves the integrity of the speech representations.

Insights and Future Directions

The success of RobustDistiller suggests that standard SSL models are often "lazy"; they do not naturally learn to ignore noise unless specifically incentivized to do so during the distillation process. By forcing the student to reconstruct clean audio from noisy inputs, the model develops a more robust understanding of speech.

However, some limitations remain. The current distillation process focuses on specific layers (the 4th, 8th, and 12th), which may not be optimal for every architecture. Additionally, the inclusion of the denoising head and online augmentation increases training time—taking roughly 43 hours on an A100 GPU compared to 30 hours for standard augmentation. Future work will likely explore whether these robustness gains hold up for more complex tasks like full-scale Automatic Speech Recognition (ASR) and if the layer selection process can be further optimized.

Summary updated

BibTeX
@inproceedings{guimaraes2023robustdistiller,
  author = {Heitor R. Guimarães and Arthur Pimentel and Anderson R. Avila and Mehdi Rezagholizadeh and Boxing Chen and Tiago H. Falk},
  title = {{Robustdistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095480},
}
Figure