Semi-Supervised Source Localization with Deep Generative Modeling

Michael J. Bianco, Sharon Gannot, Peter Gerstoft

TL;DR

This paper introduces a semi-supervised learning approach for acoustic source localization using Variational Autoencoders (VAEs). By training on a mix of a few labeled samples and many unlabeled ones, the model learns the underlying physics of reverberant environments, significantly outperforming standard supervised methods and classical algorithms when labeled data is scarce.

What Problem Does This Paper Address?

Locating a sound source (Direction of Arrival, or DOA) is notoriously difficult in "reverberant" environments—rooms where sound bounces off walls and objects, creating complex echoes. While Deep Learning has shown great promise in solving this, it typically requires massive amounts of labeled data (audio recordings paired with exact coordinates).

In the real world, collecting labeled data for every possible room configuration is expensive and impractical. However, collecting unlabeled audio is easy. The authors address the "label-limited" challenge: how can we build a robust localizer when we only have a handful of known positions but plenty of raw, unlabeled recordings?

Methods

The authors employ a Semi-supervised Variational Autoencoder (VAE-SSL), specifically based on the "M2" generative model architecture.

Key components include:

  • Feature Extraction: Instead of raw audio, they use the phase of the Relative Transfer Function (RTF) between two microphones. RTFs are useful because they represent the acoustic path while being mostly independent of the specific sound (e.g., speech vs. noise) being made.
  • The Generative Twist: Unlike a standard classifier that just maps input to a label, this VAE is trained to generate the RTF phase. By learning to reconstruct the input from a latent space, the model implicitly learns the "physics" of the room's acoustics from unlabeled data.
  • Parallel Training: The system consists of three networks: a classifier (predicts DOA), an encoder (maps input to a latent distribution), and a decoder (reconstructs the input). For labeled data, it minimizes classification error; for unlabeled data, it maximizes the "Evidence Lower Bound" (ELBO), forcing the model to find a structure in the data that makes sense even without a label.

Datasets

The study uses simulated acoustic data generated via a Room Impulse Response (RIR) generator to create four distinct scenarios:

  1. Design Case: A 6x6x2.4m room with a 500ms reverberation time ($RT_{60}$).
  2. Validation Case: Same room, but "muddier" acoustics ($RT_{60} = 700ms$).
  3. Test Case I: Same as Design, but with small (5mm) physical displacements in microphone positions to test robustness.
  4. Test Case II: A combination of mic displacement and increased reverberation ($RT_{60} = 600ms$).

The models were tested with varying numbers of labels ($J$), ranging from just 37 (one label per DOA candidate) to 999.

Key Insights and Ideas

  • Unlabeled Data as a Teacher: The core "aha" moment is that the VAE's reconstruction task acts as a regularizer. By forcing the model to understand how to "re-create" an acoustic frame, it learns features that are physically meaningful, which then makes the classification task much easier even with very few labels.
  • Robustness to Perturbations: The VAE-SSL approach proved more resilient than standard CNNs when the environment changed slightly (like moving a microphone a few millimeters). This suggests the generative process captures more general acoustic properties than a simple discriminative model.
  • The "Two-Sample" Threshold: The authors found that having as few as two labeled samples per DOA ($J=74$) was enough for the VAE-SSL to consistently outperform the classical SRP-PHAT algorithm.

Key Results

  • Superiority in Scarcity: With only 37 labels (1 per DOA), VAE-SSL achieved a Mean Absolute Error (MAE) of 27.2°, while a standard supervised CNN struggled at 39.5°.
  • Beating the Classics: In the "Test Case II" scenario with 148 labels, VAE-SSL achieved an MAE of 16.0°, outperforming the widely-used SRP-PHAT algorithm, which sat at 22.0°.
  • Crossover Point: As expected, when labels are abundant ($J=999$), a fully supervised CNN eventually wins (0.34° vs 2.24° MAE). However, the VAE-SSL remains highly competitive, proving it doesn't "sacrifice" much performance to gain its semi-supervised advantages.

Limitations and Future Work

  • Simulation vs. Reality: The study relies on simulated RIRs. While simulation is a standard starting point, real-world rooms have non-parallel walls, varying surface textures, and dynamic background noise that might challenge the VAE's generative capacity.
  • Single Source Constraint: The current model assumes only one active source at a time. Extending this to multi-source scenarios (the "cocktail party problem") would be a significant next step.
  • Latent Space Complexity: The authors used a small latent dimension ($M=2$). Exploring higher-dimensional latent spaces might allow the model to capture more complex environmental variables like source distance or room volume.

Summary updated

Abstract

We propose a semi-supervised localization approach based on deep generative modeling with variational autoencoders (VAE). Localization in reverberant environments remains a challenge, which machine learning (ML) has shown promise in addressing. Even with large data volumes, the number of labels available for supervised learning in reverberant environments is usually small. We address this issue by perform semi-supervised learning (SSL) with convolutional VAEs. The VAE is trained to generate the phase of relative transfer functions (RTFs), in parallel with a DOA classifier, on both labeled and unlabeled RTF samples. The VAE-SSL approach is compared with SRP-PHAT and fully-supervised CNNs. We find that VAE-SLL can outperform both SRP-PHAT and CNN in label-limited scenarios.
BibTeX
@inproceedings{bianco2020semi,
  author = {Michael J. Bianco and Sharon Gannot and Peter Gerstoft},
  title = {{Semi-Supervised Source Localization with Deep Generative Modeling}},
  booktitle = {MLSP},
  year = {2020},
  url = {https://openreview.net/forum?id=TK2IOb4hIh1},
}

Loading related content...

Figure