Evaluation of Deep Audio Representations for Hearables

Fabian Gröger, Pascal Baumann, Ludovic Amruthalingam, Laurent Simon, Ruksana Giurda, Simone Lionetti

Abstract

Modern hearables, such as hearing aids and smart earbuds, require a sophisticated understanding of complex acoustic environments to provide effective signal processing. However, existing audio benchmarks often focus on high-level speech or general audio classification, neglecting technical acoustic properties and suffering from potential data leakage. This paper introduces the Deep Evaluation of Audio Representations (DEAR) benchmark, a novel dataset designed to evaluate how well audio foundation models capture properties like reverberation and signal-to-noise ratio. Our evaluation of four state-of-the-art models reveals that the BEATs model, trained on diverse acoustic data, significantly outperforms speech-centric models, establishing a new baseline for hearable-focused AI.

Introduction

The evolution of "hearables"—a category encompassing medical hearing aids and consumer-grade noise-canceling earbuds—has shifted the focus of audio research toward real-time environmental awareness. These devices must perform complex tasks such as active noise cancellation (ANC), spatial perception preservation, and speech enhancement in dynamic settings. While self-supervised learning (SSL) foundation models have shown remarkable success in audio tasks, their utility for hearables remains under-explored.

Existing benchmarks like SUPERB or HEAR primarily evaluate models on speech recognition or general sound event detection. These benchmarks often overlook low-level technical acoustic properties—such as Direct-to-Reverberant energy Ratio (DRR) and reverberation time (RT60)—which are critical for the signal processing pipelines used in hearables. Furthermore, the use of public datasets in these benchmarks introduces the risk of "data leakage," where models may have encountered test data during their massive pre-training phases.

The DEAR Benchmark

To address these limitations, we introduce the Deep Evaluation of Audio Representations (DEAR) dataset. DEAR is specifically engineered to be leak-proof and focused on the technical requirements of hearable devices.

Dataset Construction

The dataset consists of 1,158 mono tracks, each 30 seconds in length. To ensure no overlap with common pre-training corpora, the authors utilized proprietary anechoic speech recordings mixed with licensed, high-quality background scenes from the HOA-SSR dataset. This controlled synthesis allows for precise labeling of acoustic parameters that are otherwise difficult to obtain in "in-the-wild" recordings.

Evaluation Tasks

The benchmark evaluates models across eight downstream tasks, categorized into three domains:

  1. Contextual Awareness: Identifying environment types (e.g., domestic vs. nature), indoor/outdoor settings, and the nature of noise (stationary vs. transient).
  2. Source Analysis: Detecting speech presence and performing speaker counting.
  3. Acoustic Properties: Estimating technical metrics including DRR, RT60, and Signal-to-Noise Ratio (SNR).

Experimental Methodology

We evaluated four prominent SSL audio foundation models: Wav2Vec2.0, HuBERT, WavLM, and BEATs. These models vary in their pre-training objectives and data, ranging from speech-only (Wav2Vec2.0) to diverse audio (BEATs).

To assess the quality of the learned representations, we employed a "frozen" probing strategy. The pre-trained weights of the models were kept static, and simple linear or k-nearest neighbor (kNN) probes were trained on the extracted features. This approach ensures that the performance reflects the intrinsic information captured by the model during pre-training rather than the model's ability to learn a new task through extensive fine-tuning.

Results and Discussion

The experimental results highlight a clear hierarchy in model performance, with the BEATs model emerging as the dominant architecture across all categories.

The Superiority of Diverse Training

BEATs consistently outperformed models like Wav2Vec2.0, HuBERT, and WavLM. While the latter three are primarily trained on speech-centric datasets (e.g., LibriSpeech), BEATs is trained on AudioSet, which contains a vast array of environmental sounds. Interestingly, BEATs' superiority extended even to speech-related tasks within the DEAR dataset, suggesting that exposure to diverse acoustic contexts enhances a model's ability to process speech in noisy, real-world environments.

Capturing Technical Acoustics

One of the most significant findings is the emergent ability of BEATs to encode technical acoustic properties. In tasks involving RT60 and DRR, BEATs outperformed its competitors by as much as 30 percentage points. This suggests that its unique acoustic tokenizer objective allows it to internalize the physics of sound—such as how reverberation affects a signal—without explicit supervision.

Challenges in Speaker Counting and DRR

Despite the overall success of foundation models, certain tasks remain challenging. Speaker counting in complex, noisy environments proved difficult for all models, indicating a need for better spatial or temporal resolution in representations. Similarly, estimating DRR remains a high-difficulty task, representing a clear area for future algorithmic development.

Conclusion

The DEAR benchmark provides a rigorous framework for evaluating audio representations through the lens of hearable technology. Our findings underscore the importance of diverse pre-training data and suggest that models like BEATs offer a strong foundation for next-generation audio processing.

However, a significant hurdle remains: deployment. With approximately 90 million parameters, models like BEATs are currently too computationally expensive for the strict power and latency requirements of hearing aids and earbuds. Future research must focus on model compression and the development of "tiny" foundation models that can bring these advanced acoustic insights to edge devices, ultimately improving the auditory experience for millions of users.

Summary updated

BibTeX
@inproceedings{groger2025evaluation,
  author = {Fabian Gröger and Pascal Baumann and Ludovic Amruthalingam and Laurent Simon and Ruksana Giurda and Simone Lionetti},
  title = {{Evaluation of Deep Audio Representations for Hearables}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10887737},
}

Loading related content...

Figure