Unsupervised Training of Neural Mask-Based Beamforming

Lukas Drude, Jahn Heymann, Reinhold Haeb-UmbachInterspeech 2019

TL;DR

This paper introduces a novel unsupervised training method for neural network-based beamforming systems. By directly optimizing a likelihood criterion derived from a spatial mixture model, the approach eliminates the need for paired clean and noisy speech data, enabling training on real-world recordings and outperforming previous unsupervised methods.

What Problem Does This Paper Address?

Multi-channel audio processing, particularly beamforming, is crucial for robust automatic speech recognition (ASR) in noisy environments, especially for far-field applications like smart assistants. Traditional beamforming relies on estimating spatial covariance matrices of speech and noise. While neural networks have shown promise in estimating these masks, supervised training requires paired clean and noisy audio, which is often impractical to obtain for real-world scenarios (e.g., Lombard effect) and expensive to simulate accurately. Existing unsupervised methods often rely on teacher models or complex training procedures. This paper addresses the need for an unsupervised training approach that is simpler, more efficient, and can leverage abundant real-world noisy data without requiring clean targets.

Methods

The core of the proposed method is an unsupervised training approach for a neural mask estimator. Instead of relying on supervised targets, the network is trained to maximize a likelihood criterion derived from a spatial mixture model. Specifically:

  • Spatial Mixture Model: The paper utilizes a probabilistic spatial mixture model, specifically a complex angular central Gaussian mixture model (CACGMM), to model the multi-channel observations. This model assumes that each time-frequency (tf)-bin of the observed signal can belong to one of several classes (speech or noise).
  • Unsupervised Training Loop: The training involves an iterative process:
  • Initialization: The neural mask estimator is initialized randomly.
  • M-step: The current network output (class affiliations) is used to estimate the parameters of the spatial mixture model (prior probabilities $\pi_{kf}$ and covariance matrices $B_{kf}$).
  • E-step: The likelihood of the observed data is calculated using the estimated model parameters.
  • Gradient Backpropagation: The gradient of this likelihood is backpropagated through the Expectation-Maximization (EM) algorithm's update equations and into the neural mask estimator. This allows the network's parameters to be updated directly based on how well they contribute to the observed data's likelihood under the spatial model.
  • Key Innovation: The paper highlights that a single EM iteration per training step is sufficient, and the spatial model is integrated into the processing graph, benefiting from the most recent network parameters for initialization. This is a significant simplification compared to previous methods that might require many EM iterations.
  • Permutation Alignment: To address the potential frequency permutation problem (where the same class might be assigned different indices across frequency bins), a permutation alignment algorithm is employed.

Datasets

  • CHiME 4: Used for training and evaluation. This dataset consists of six-channel recordings from microphones on a tablet, obtained in public spaces, and is primarily degraded by noise.
  • REVERB: Used for evaluation. This dataset features eight-channel recordings in a reverberant enclosure, making it suitable for evaluating performance in reverberation-dominated scenarios.

Key Insights and Ideas

  • Direct Likelihood Optimization: The most significant insight is that directly optimizing the likelihood of the observed data under a spatial mixture model, using gradients from the EM algorithm, provides an effective unsupervised training signal for neural mask estimators.
  • Eliminating Clean Data Dependency: This approach completely removes the need for paired clean and noisy data, opening the door to training on vast amounts of readily available real-world noisy recordings.
  • Efficiency of Single EM Iteration: The finding that a single EM iteration per training step is sufficient is a key practical advantage, significantly reducing computational complexity compared to methods requiring multiple EM iterations.
  • Integration of Spatial Model: Integrating the spatial mixture model directly into the training graph, with its parameters being updated based on the network's output, ensures a more adaptive and potentially better-initialized model.
  • Competitive Performance: The unsupervised system achieves performance comparable to supervised systems, demonstrating the viability of this approach.

Key Results

  • CHiME 4 Performance: The proposed unsupervised system achieved a Word Error Rate (WER) of 7.80% on the CHiME 4 real test set, which is competitive with supervised systems (e.g., 7.71% with oracle masks and sigmoid output).
  • REVERB Performance: On the REVERB dataset, the unsupervised system achieved a WER of 7.02% (with WPE) and 7.86% (without WPE), demonstrating strong performance even in reverberant conditions and outperforming a teacher-based unsupervised system.
  • Comparison to Teacher-Based Methods: The proposed method is shown to be on par with or better than unsupervised methods that rely on a pre-trained teacher model.
  • Benefits of Additional EM Step: An additional EM step at inference time generally improved performance across various loss functions.

Limitations and Future Work

  • Permutation Problem: While addressed by permutation alignment, the frequency permutation problem can still introduce challenges, especially early in training, leading to conflicting gradients.
  • Computational Cost: While more efficient than some unsupervised methods, the EM algorithm and backpropagation through it still incur computational costs.
  • Future Work: The authors suggest exploring this training scheme for multi-speaker scenarios and single-channel speech enhancement, indicating broader applicability beyond the current beamforming focus. They also note the connection to variational autoencoders and suggest further research into better uncertainty estimation for masks and covariance matrices.

Summary updated

BibTeX
@inproceedings{drude2019unsupervised,
  author = {Lukas Drude and Jahn Heymann and Reinhold Haeb-Umbach},
  title = {{Unsupervised Training of Neural Mask-Based Beamforming}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure