An Exploration into the Performance of Unsupervised Cross-Task Speech Representations for "In the Wild" Edge Applications

Heitor R. Guimarães, Arthur Pimentel, Anderson R. Avila, Mehdi Rezagholizadeh, Tiago H. FalkarXiv 2023 · DOI · arXiv

Abstract

Self-supervised learning (SSL) has revolutionized speech processing by enabling models to learn powerful representations from unlabeled data. However, the transition of these models from controlled laboratory environments to "in the wild" edge applications—such as smart assistants and wearables—presents significant challenges. This paper investigates the performance of the HuBERT architecture across various scales and environmental conditions. We evaluate the trade-offs between model size, task-specific performance, and robustness to additive noise and reverberation. Our findings reveal a "reverb wall" that severely degrades performance across all model sizes. Furthermore, we demonstrate that domain adversarial training in smaller models (Robust-HuBERT) can match or exceed the performance of much larger models in noisy conditions, offering a viable path for resource-constrained edge deployment.

Introduction

The advent of self-supervised learning (SSL) has shifted the paradigm in speech technology. Models like HuBERT (Hidden-unit BERT) have demonstrated state-of-the-art performance on benchmarks like SUPERB by leveraging massive amounts of unlabeled audio. Despite these successes, most evaluations occur under "clean" conditions that do not reflect the reality of edge computing. Edge devices operate in unpredictable environments characterized by background noise (additive noise) and room echoes (convolutive noise or reverberation).

Moreover, edge hardware imposes strict constraints on memory and computational power. While scaling models to billions of parameters often improves robustness, such models are unsuitable for deployment on devices like smartwatches or low-power sensors. This study addresses this gap by exploring how different HuBERT variants handle environmental degradation and whether architectural optimizations can compensate for reduced parameter counts.

Methodology

The researchers evaluated four distinct versions of the HuBERT architecture to understand the impact of scale and training techniques:

  • HuBERT Base (95M params): Trained on 960 hours of Librispeech.
  • HuBERT Large (317M params): Trained on 60,000 hours of Libri-light.
  • HuBERT X-Large (964M params): Also trained on 60,000 hours of Libri-light.
  • Robust-HuBERT (95M params): A variant of the Base model utilizing domain adversarial training. This approach employs a discriminator to identify noise sources, forcing the encoder to learn features invariant to environmental distortions.

The models were tested on three downstream tasks representing different speech processing domains: Keyword Spotting (KWS) for content, Intent Classification (IC) for semantics, and Emotion Recognition (ER) for para-linguistic features. To simulate real-world conditions, clean test data was modified using background noise from the DNS Challenge 4 and Room Impulse Responses (RIRs) to simulate reverberation.

Results and Discussion

The experimental results, summarized in Figure 1, highlight a stark contrast between performance in clean versus degraded environments.

Table 1: Performance comparison of HuBERT variants across different tasks and noise conditions. (c) denotes clean, (n) noisy, and (n+r) noise plus reverberation.

The "Reverb Wall"

One of the most significant findings is the devastating impact of reverberation. While models often maintained reasonable accuracy under simple additive noise, the introduction of reverberation (the "n+r" condition) caused performance to plummet. For instance, the HuBERT Base model's accuracy in Keyword Spotting dropped from 96.3% (clean) to 52.9% (noise + reverb). This "reverb wall" suggests that current SSL representations struggle to decouple the speech signal from the acoustic properties of the environment.

Efficiency vs. Robustness

The data confirms that larger models generally offer better inherent robustness. The HuBERT X-Large model consistently outperformed the Base model in most degraded scenarios. However, the performance gap between the 95M parameter Base model and the nearly 1B parameter X-Large model is often not large enough to justify the 10x increase in resource requirements for edge applications.

The Power of Adversarial Training

The Robust-HuBERT model provides a compelling alternative to brute-force scaling. Despite having the same parameter count as the Base model, Robust-HuBERT achieved 91.4% accuracy in noisy Keyword Spotting, significantly outperforming the X-Large model's 87.4%. This indicates that targeted training strategies can effectively bake robustness into smaller, edge-friendly architectures.

Task-Specific Sensitivity

The impact of noise varied across tasks. Intent Classification appeared the most resilient, likely because semantic meaning can often be inferred even from a degraded signal. In contrast, Emotion Recognition was the most fragile; in the "n+r" condition, accuracy for all models fell below 50%, highlighting the difficulty of preserving fine-grained para-linguistic features in the presence of echoes.

Conclusion

This exploration underscores the challenges of deploying SSL speech models "in the wild." While massive models provide a degree of natural robustness, they remain impractical for the edge. The success of Robust-HuBERT suggests that the path forward lies in algorithmic innovations like domain adversarial training rather than simple parameter scaling. Future work must prioritize overcoming the "reverb wall" and developing models that are both computationally efficient and environmentally resilient to truly enable the next generation of edge-based speech AI.

Summary updated

BibTeX
@misc{guimaraes2023exploration,
  author = {Heitor R. Guimarães and Arthur Pimentel and Anderson R. Avila and Mehdi Rezagholizadeh and Tiago H. Falk},
  title = {{An Exploration into the Performance of Unsupervised Cross-Task Speech Representations for "In the Wild" Edge Applications}},
  year = {2023},
  eprint = {2305.05443},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.05443},
}
Figure