Abstract
Continuous speech separation using a microphone array was shown to be promising in dealing with the speech overlap problem in natural conversation transcription. This paper proposes VarArray, an array-geometry-agnostic speech separation neural network model. The proposed model is applicable to any number of microphones without retraining while leveraging the nonlinear correlation between the input channels. The proposed method adapts different elements that were proposed before separately, including transform-average-concatenate, conformer speech separation, and inter-channel phase differences, and combines them in an efficient and cohesive way. Large-scale evaluation was performed with two real meeting transcription tasks by using a fully developed transcription system requiring no prior knowledge such as reference segmentations, which allowed us to measure the impact that the continuous speech separation system could have in realistic settings. The proposed model outperformed a previous approach to array-geometry-agnostic modeling for all of the geometry configurations considered, achieving asclite-based speaker-agnostic word error rates of 17.5% and 20.4% for the AMI development and evaluation sets, respectively, in the end-to-end setting using no ground-truth segmentations.
Introduction
In natural conversations, overlapping speech presents a significant challenge for Automatic Speech Recognition (ASR) systems. Continuous Speech Separation (CSS) addresses this by "unmixing" voices into clean, separate audio streams. While multi-microphone setups provide spatial data to aid this process, most models are "geometry-dependent," meaning they are optimized for a specific microphone layout. This lack of flexibility limits their deployment across different devices. Previous "geometry-agnostic" attempts, such as Split-Apply-Combine (SAC), often fail to capture complex non-linear correlations between channels or become computationally prohibitive as the number of microphones increases.
Methods
VarArray is designed to be both geometry-agnostic and computationally efficient. Its architecture is built upon three primary components:
- Invariant Feature Design: The model utilizes the magnitude of the average spectrum across all microphones and the Inter-channel Phase Difference (IPD) relative to that average. This ensures the input representation remains consistent regardless of microphone indexing.
- Conformer + TAC Architecture: The model employs Conformer blocks to capture temporal speech patterns. These are interleaved with Transform-Average-Concatenate (TAC) layers, which pool information across all available microphones in a permutation-invariant manner. This allows the model to fuse spatial data without prior knowledge of the array geometry.
- Early Stream Merging: To optimize computational efficiency, VarArray merges multiple feature streams into a single representative stream early in the network (after the third Conformer block). This significantly reduces the computational load for arrays with high microphone counts.
The system is further refined through End-to-End (E2E) optimization, fine-tuning the separation model with an ASR-based loss function to ensure the output is optimized for transcription accuracy rather than just human audibility.
Evaluation and Results
The researchers evaluated VarArray on two real-world meeting corpora: the AMI-MDM benchmark (8-channel circular array) and Microsoft Internal Meetings (7-channel array). Training utilized a 438-hour simulated dataset covering diverse acoustic environments and geometries.
VarArray consistently outperformed baseline systems, including the SAC approach and fixed-geometry models. Notably, it maintained superior performance even when the number of microphones was reduced (e.g., from 8 to 4 for AMI), demonstrating significant robustness to varying or degraded hardware configurations.
Key findings include:
- Spatial Generalization: Training on diverse simulated arrays allows the model to learn general spatial filtering principles, often outperforming models trained on a single fixed geometry.
- Efficiency: Early fusion of microphone streams maintains performance while reducing latency and power consumption.
- ASR-Centric Optimization: Fine-tuning with Word Error Rate (WER) loss proved more effective than signal-level losses, prioritizing features critical for transcription.
Conclusion
VarArray represents a significant step toward universal, "plug-and-play" speech separation for meeting transcription. It achieves state-of-the-art results on the AMI dataset without requiring ground-truth segmentations. Future work will focus on expanding the model to handle more than two simultaneous speakers and applying the framework to tasks like personalized noise reduction in diverse acoustic environments.
Summary updated