Semi-supervised source localization with deep generative modeling

Michael J. Bianco, Sharon Gannot, Peter Gerstoft

Abstract

We propose a semi-supervised localization approach based on deep generative modeling with variational autoencoders (VAEs). Localization in reverberant environments remains a challenge, which machine learning (ML) has shown promise in addressing. Even with large data volumes, the number of labels available for supervised learning in reverberant environments is usually small. We address this issue by performing semi-supervised learning (SSL) with convolutional VAEs. The VAE is trained to generate the phase of relative transfer functions (RTFs), in parallel with a DOA classifier, on both labeled and unlabeled RTF samples. The VAE-SSL approach is compared with SRP-PHAT and fully-supervised CNNs. We find that VAE-SSL can outperform both SRP-PHAT and CNN in label-limited scenarios.

Introduction

Locating a sound source, or determining its Direction of Arrival (DOA), is a fundamental task in acoustic signal processing. However, it is notoriously difficult in reverberant environments where sound reflections from walls and objects create complex echoes. While deep learning has shown significant promise in overcoming these challenges, standard supervised models require massive amounts of labeled data—audio recordings paired with precise ground-truth coordinates.

In practical scenarios, collecting labeled data for every possible room configuration is expensive and often impossible. Conversely, raw, unlabeled audio is abundant. This paper addresses the "label-limited" challenge: how to build a robust localizer using a small set of labeled samples augmented by a large volume of unlabeled recordings.

Methodology

The authors propose a Semi-supervised Variational Autoencoder (VAE-SSL) framework based on the "M2" generative model architecture. The core idea is to leverage the generative capacity of VAEs to learn the underlying physics of acoustic environments from unlabeled data.

Feature Extraction and Model Architecture

Instead of raw audio, the model utilizes the phase of the Relative Transfer Function (RTF) between two microphones. RTFs are advantageous because they characterize the acoustic path while remaining largely independent of the specific source signal (e.g., speech vs. noise).

Graphical models for the semi-supervised VAE. (a) Inference model for the latent variable z. (b) Generative model for the observation x and label y. (c) Generative model for the observation x given y and z.

As shown in the graphical models above, the system integrates three networks:

  1. Classifier: Predicts the DOA label $y$ from the input $x$.
  2. Encoder: Maps the input $x$ and label $y$ to a latent distribution $z$.
  3. Decoder: Reconstructs the input $x$ from the latent space $z$ and label $y$.

For labeled data, the model minimizes classification error. For unlabeled data, it maximizes the Evidence Lower Bound (ELBO), forcing the latent space to capture meaningful acoustic structures even without explicit labels.

Experimental Setup

The study utilizes simulated acoustic data across four distinct scenarios to test robustness:

  • Design Case: A standard 6x6x2.4m room ($RT_{60} = 500ms$).
  • Validation Case: A more reverberant version of the design room ($RT_{60} = 700ms$).
  • Test Case I: The design room with 5mm physical displacements in microphone positions.
  • Test Case II: A combination of mic displacement and increased reverberation ($RT_{60} = 600ms$).

The models were evaluated with varying numbers of labels ($J$), ranging from 37 (one label per DOA candidate) to 999.

Key Results and Discussion

The VAE-SSL approach demonstrates significant advantages in scenarios where labeled data is scarce. By using the VAE's reconstruction task as a regularizer, the model learns physically meaningful features that aid classification.

Performance comparison (MAE and Accuracy) for the Design Case across different label counts (J).

Performance Highlights

  • Superiority in Scarcity: With only 37 labels (1 per DOA), VAE-SSL achieved a Mean Absolute Error (MAE) of 27.2°, significantly outperforming a standard supervised CNN which struggled at 39.5°.
  • Beating Classical Algorithms: In Test Case II with 148 labels, VAE-SSL achieved an MAE of 16.0°, surpassing the widely-used SRP-PHAT algorithm (22.0°).
  • Robustness: The VAE-SSL proved more resilient than standard CNNs to small physical perturbations, such as microphone displacement, suggesting the generative process captures more general acoustic properties.
  • Crossover Point: While a fully supervised CNN eventually wins when labels are abundant ($J=999$, 0.34° vs 2.24° MAE), the VAE-SSL remains highly competitive, showing it does not sacrifice significant performance to gain its semi-supervised benefits.

Conclusion

This research demonstrates that semi-supervised deep generative modeling is a powerful tool for acoustic source localization in reverberant environments. By effectively utilizing unlabeled data, the VAE-SSL framework provides a robust solution for real-world applications where ground-truth labels are difficult to obtain. Future work will explore extending this model to multi-source scenarios and higher-dimensional latent spaces to capture more complex environmental variables.

Summary updated

Abstract

We propose a semi-supervised localization approach based on deep generative modeling with variational autoencoders (VAEs). Localization in reverberant environments remains a challenge, which machine learning (ML) has shown promise in addressing. Even with large data volumes, the number of labels available for supervised learning in reverberant environments is usually small. We address this issue by performing semi-supervised learning (SSL) with convolutional VAEs. The VAE is trained to generate the phase of relative transfer functions (RTFs), in parallel with a DOA classifier, on both labeled and unlabeled RTF samples. The VAE-SSL approach is compared with SRP-PHAT and fully-supervised CNNs. We find that VAE-SSL can outperform both SRP-PHAT and CNN in label-limited scenarios.
BibTeX
@inproceedings{bianco2020semi,
  author = {Michael J. Bianco and Sharon Gannot and Peter Gerstoft},
  title = {{Semi-supervised source localization with deep generative modeling}},
  booktitle = {MLSP},
  year = {2020},
  url = {https://openreview.net/forum?id=TK2IOb4hIh1},
}
Figure