Neural Network-Based Virtual Microphone Estimator

Tsubasa Ochiai, Marc Delcroix, Tomohiro Nakatani, Rintaro Ikeshita, Keisuke Kinoshita, Shoko Araki

TL;DR

This paper introduces a novel approach, the Neural Network-Based Virtual Microphone Estimator (NN-VME), that uses time-domain neural networks to generate virtual microphone signals. This method bypasses the need for complex physical models and can be trained directly on real-world noisy recordings, significantly improving performance in audio processing tasks like speech enhancement and recognition.

What Problem Does This Paper Address?

Developing advanced microphone array signal processing techniques often requires a large number of microphones to achieve high performance. However, practical constraints like device size and cost limit the number of microphones that can be deployed, especially in consumer electronics like smartphones. While previous methods attempted to virtually increase the microphone count by interpolating signals based on physical models, these assumptions often don't hold in real-world scenarios. This paper addresses the challenge of effectively augmenting microphone arrays with virtual microphones in a way that is robust to real-world conditions and doesn't rely on potentially inaccurate physical models.

Methods

The core of this paper is the Neural Network-Based Virtual Microphone Estimator (NN-VME). This method employs a time-domain neural network architecture, inspired by Conv-TasNet, to directly estimate virtual microphone signals in the time domain.

Here's a breakdown of the key methodological aspects:

  • Time-Domain Neural Network: Unlike traditional methods that might operate in the frequency domain, NN-VME works directly with raw audio waveforms. This allows it to capture fine temporal details and potentially preserve phase information crucial for spatial processing.
  • Supervised Learning Framework: The NN-VME is trained using a supervised learning approach. This means that during training, the network is provided with both the input real microphone signals and the desired "ground truth" signals from the locations of the virtual microphones.
  • No Physical Model Assumptions: A significant departure from prior work, NN-VME does not rely on explicit physical models of sound propagation (e.g., plane wave assumptions). This makes it more adaptable to diverse and complex acoustic environments.
  • Training on Real Recordings: The supervised learning framework allows the NN-VME to be trained directly on multi-channel real recordings, avoiding the need for simulated data or perfectly clean signals. This is a major advantage for practical deployment.
  • Application to Beamforming: The estimated virtual microphone signals are then integrated with existing array processing techniques, specifically a frequency-domain Minimum Variance Distortionless Response (MVDR) beamformer. This augmented beamformer aims to improve speech enhancement and recognition.
  • Virtual Microphone Loading: To mitigate potential artifacts introduced by the virtual microphone estimation and reduce its influence on the beamformer, a "virtual microphone loading" term is introduced into the noise spatial covariance matrix. This term effectively down-weights the contribution of virtual microphones if they are perceived to be contaminated by noise.

Datasets

  • CHiME-4 Corpus: This corpus was used for both training and evaluation. It features real recordings from noisy public environments captured by a rectangular microphone array with 6 channels. The corpus includes both simulated and real speech data.
  • Training Data: Consisted of 3 hours of real speech data from 4 speakers and 15 hours of simulated speech data from 83 speakers.
  • Evaluation Data: Initially comprised 1320 utterances. However, utterances with microphone failures (affecting channels 4 and 5) were excluded, resulting in 1149 evaluated utterances.

Key Insights and Ideas

  • Bridging the Gap Between Real and Virtual: The NN-VME successfully bridges the gap between the limited number of physical microphones and the desired number for enhanced array processing, doing so without relying on potentially brittle physical models.
  • Power of Time-Domain Networks for Spatial Audio: The success of NN-VME highlights the growing capability of time-domain neural networks not just for source separation but also for extracting and utilizing spatial information from microphone arrays.
  • Direct Training on Real Data is Feasible: The ability to train the NN-VME directly on real, noisy multi-channel recordings is a significant practical advantage, making the approach more accessible and robust than methods requiring perfect clean signals or extensive simulations.
  • Virtual Microphone Loading Improves Robustness: The introduction of the virtual microphone loading term demonstrates a clever way to control the influence of potentially less reliable virtual microphone signals, leading to improved performance, particularly in ASR tasks.

Key Results

  • Virtual Microphone Estimation Performance: The NN-VME achieved high Signal-to-Distortion Ratio (SDR) scores (over 12 dB) for estimating virtual microphone signals, even when interpolating or extrapolating their positions relative to the real microphones. These scores were significantly higher than using the closest real microphone as a reference.
  • Beamformer Enhancement Performance: When combined with an MVDR beamformer, the NN-VME (VM BF) showed improved performance over a beamformer using only real microphones (RM BF).
  • On simulated data, the VM BF achieved higher SDR scores.
  • On real data, the VM BF reduced the Word Error Rate (WER) for Automatic Speech Recognition (ASR) by up to 0.9% compared to the RM BF.
  • Effectiveness of Virtual Microphone Loading: The virtual microphone loading technique further improved ASR performance, reducing WER by approximately 1.5% in some configurations.

Limitations and Future Work

  • Limitations:
    • The evaluation of the beamformer's enhancement performance (using clean signals as reference) was only possible on simulated data due to the requirement of clean reference signals.
  • The paper notes that increasing the number of input channels did not always lead to significant improvements in virtual microphone estimation performance, suggesting potential for further investigation.
  • The study excluded utterances with microphone failures, which is a common issue in real-world recordings.
  • Future Work:
    • Evaluating the NN-VME's effectiveness across a wider variety of microphone array configurations.
  • Testing the method in diverse acoustic conditions.
  • Exploring its application with other array processing techniques, such as source separation and source localization.

Summary updated

Abstract

Developing microphone array technologies for a small number of microphones is important due to the constraints of many devices. One direction to address this situation consists of virtually augmenting the number of microphone signals, e.g., based on several physical model assumptions. However, such assumptions are not necessarily met in realistic conditions. In this paper, as an alternative approach, we propose a neural network-based virtual microphone estimator (NN-VME). The NN-VME estimates virtual microphone signals directly in the time domain, by utilizing the precise estimation capability of the recent time-domain neural networks. We adopt a fully supervised learning framework that uses actual observations at the locations of the virtual microphones at training time. Consequently, the NN-VME can be trained using only multi-channel observations and thus directly on real recordings, avoiding the need for unrealistic physical model-based assumptions. Experiments on the CHiME-4 corpus show that the proposed NN-VME achieves high virtual microphone estimation performance even for real recordings and that a beamformer augmented with the NN-VME improves both the speech enhancement and recognition performance.
BibTeX
@inproceedings{ochiai2021neural,
  author = {Tsubasa Ochiai and Marc Delcroix and Tomohiro Nakatani and Rintaro Ikeshita and Keisuke Kinoshita and Shoko Araki},
  title = {{Neural Network-Based Virtual Microphone Estimator}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9415121},
}
Figure