VarArray: Array-Geometry-Agnostic Continuous Speech Separation

Takuya Yoshioka, Xiaofei Wang, Dongmei Wang, Min Tang, Zirun Zhu, Zhuo Chen, Naoyuki Kanda

TL;DR

VarArray is a novel neural network architecture for continuous speech separation that works with any microphone array geometry without needing to be retrained. By combining Conformer blocks with Transform-Average-Concatenate (TAC) layers, it effectively leverages spatial information across varying numbers of microphones, achieving state-of-the-art results on real-world meeting transcription tasks.

What Problem Does This Paper Address?

In natural conversations, people often talk over one another. To transcribe these meetings accurately, researchers use Continuous Speech Separation (CSS) to "unmix" overlapping voices into separate, clean audio streams before feeding them into an Automatic Speech Recognition (ASR) system.

While multi-microphone setups provide rich spatial data to help this unmixing, most existing models are "geometry-dependent"—they are hard-coded for a specific microphone layout (e.g., a 7-mic circular array). If you move the software to a device with a different layout, the model fails. Previous "geometry-agnostic" attempts, like the Split-Apply-Combine (SAC) approach, process each channel independently and average the results. However, SAC is computationally expensive as the number of mics grows and fails to capture complex, non-linear correlations between channels.

Methods

The authors introduce VarArray, designed to be both geometry-agnostic and computationally efficient. Its architecture rests on three pillars:

  1. Invariant Feature Design: Instead of feeding the model raw data from every mic, they use the magnitude of the average spectrum across all mics and the Inter-channel Phase Difference (IPD) relative to that average. This ensures the input looks the same regardless of how the microphones are indexed.
  2. Conformer + TAC Architecture: The model uses Conformer blocks to capture temporal patterns in speech. Interspersed between these are Transform-Average-Concatenate (TAC) layers. TAC is a cross-channel module that pools information from all available microphones in a permutation-invariant way, allowing the model to "fuse" spatial data without knowing the specific array geometry.
  3. Early Stream Merging: To keep computational costs low, VarArray doesn't process every microphone stream through the entire network. Instead, it merges the multiple feature streams into a single stream early in the network (after the third Conformer block), significantly reducing the math required for arrays with many microphones.

The system is further refined using End-to-End (E2E) optimization, where the separation model is fine-tuned using an ASR-based loss function. This ensures the separated audio isn't just "clean" to the human ear, but optimized for the ASR engine's accuracy.

Datasets

The researchers evaluated VarArray on two challenging real-world meeting corpora:

  • AMI (AMI-MDM): A standard public benchmark using an 8-channel circular microphone array.
  • Microsoft Internal Meetings (MS): A collection of real meetings recorded with a 7-channel array (6 in a circle, 1 in the center).

For training, they used a massive 438-hour simulated dataset created by mixing clean speech (WSJ SI-284) with various room impulse responses (RIRs) and noise profiles to simulate different acoustic environments and microphone geometries.

Key Insights and Ideas

  • Spatial Generalization: The most striking finding is that a geometry-agnostic model can actually outperform models trained specifically for a single geometry. By training on a diverse set of simulated arrays, the model learns general principles of spatial filtering rather than memorizing specific delays.
  • Efficiency via Fusion: You don't need to track every microphone's individual data through the whole deep network. Fusing the information into a single "representative" stream early on maintains performance while drastically cutting latency and power consumption.
  • ASR-Friendly Separation: Fine-tuning with an ASR loss (Word Error Rate) proved more effective than traditional signal-level losses. The model learns to prioritize the features of speech that matter most for transcription, even if the resulting audio contains some residual noise.

Key Results

VarArray consistently outperformed baseline systems across different microphone counts:

  • AMI-dev (8 mics): Achieved a Word Error Rate (WER) of 17.7%, beating the SAC baseline (18.3%) and the fixed-geometry baseline (18.8%).
  • Microsoft Meetings (7 mics): Achieved a WER of 15.5%, compared to 16.0% for the baseline.
  • Robustness: When tested on reduced setups (using only 4 mics for AMI or 3 for MS), VarArray maintained its lead, proving it can adapt to "broken" or smaller arrays on the fly without retraining.
  • State-of-the-Art: The system achieved the best-reported results on the AMI dataset under the realistic condition of having no ground-truth segmentations.

Limitations and Future Work

While VarArray is highly effective for the common "two-speaker overlap" scenario found in meetings, the authors note that the current implementation is optimized for up to two simultaneous speakers ($K=2$).

Future research directions include:

  • Expanding the model to handle more simultaneous speakers.
  • Applying the VarArray framework to other tasks like personalized noise reduction (where the model focuses on a specific "target" speaker).
  • Testing the model's performance in even more diverse acoustic environments, such as very large halls or outdoor settings.

Summary updated

Abstract

Continuous speech separation using a microphone array was shown to be promising in dealing with the speech overlap problem in natural conversation transcription. This paper proposes VarArray, an array-geometry-agnostic speech separation neural network model. The proposed model is applicable to any number of microphones without retraining while leveraging the nonlinear correlation between the input channels. The proposed method adapts different elements that were proposed before separately, including transform-average-concatenate, conformer speech separation, and inter-channel phase differences, and combines them in an efficient and cohesive way. Large-scale evaluation was performed with two real meeting transcription tasks by using a fully developed transcription system requiring no prior knowledge such as reference segmentations, which allowed us to measure the impact that the continuous speech separation system could have in realistic settings. The proposed model outperformed a previous approach to array-geometry-agnostic modeling for all of the geometry configurations considered, achieving asclite-based speaker-agnostic word error rates of 17.5% and 20.4% for the AMI development and evaluation sets, respectively, in the end-to-end setting using no ground-truth segmentations.
BibTeX
@inproceedings{yoshioka2022vararray,
  author = {Takuya Yoshioka and Xiaofei Wang and Dongmei Wang and Min Tang and Zirun Zhu and Zhuo Chen and Naoyuki Kanda},
  title = {{VarArray: Array-Geometry-Agnostic Continuous Speech Separation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=lvaYOzWGo-},
}

Loading related content...

Figure