Vararray Meets T-Sot: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition

Naoyuki Kanda, Jian Wu, Xiaofei Wang, Zhuo Chen, Jinyu Li, Takuya Yoshioka

TL;DR

Microsoft researchers have unveiled t-SOT-VA, a new framework that sets a state-of-the-art benchmark for streaming distant conversational speech recognition. By combining a geometry-agnostic front-end (VarArray) with a multi-talker back-end (t-SOT), the system achieves record-low word error rates on the AMI meeting corpus while maintaining the low latency required for real-time applications.

What Problem Does This Paper Address?

Transcribing a meeting where people are far from the microphone and frequently talk over one another is a "boss level" challenge in speech processing. This is known as Distant Conversational Speech Recognition (DCSR).

Traditional systems struggle with two main things:

  1. Acoustic Distortion: Sound bouncing off walls and the distance between the speaker and the mic.
  2. Overlapping Speech: When two or more people speak at once, standard ASR (Automatic Speech Recognition) models often collapse or only "hear" the loudest person.

Furthermore, most high-performing DCSR systems are "offline" (they need the whole recording to process it) or are hard-coded to a specific microphone arrangement. The authors sought a solution that is streaming (real-time), geometry-agnostic (works with any mic array), and highly accurate.

Methods

The authors combined two recent Microsoft innovations into a single pipeline:

  • VarArray (The Front-End): This is a "Continuous Speech Separation" (CSS) model. It takes multi-channel audio from any microphone array and separates it into two clean, overlap-free audio streams. Crucially, it doesn't need to know the microphone layout beforehand.
  • t-SOT (The Back-End): Short for token-level Serialized Output Training. Instead of having a separate output for every possible speaker, t-SOT interleaves the words from different speakers into a single sequence, using a special separator token (<cc>) to indicate when the "virtual channel" switches.

The "Secret Sauce": Two-Channel Transformer Transducer

The core novelty is a two-channel transformer transducer that takes both audio streams from VarArray simultaneously. The model uses "cross-channel transformation" (either scaling or projection) to allow the two branches of the neural network to share information.

To make the system robust, they used a three-stage training scheme:

  1. Pre-training: Training on massive amounts of single-talker data.
  2. Simulation: Training the two-channel model on simulated data that mimics common front-end errors, like "leakage" (the same voice appearing in both streams) or "utterance splits" (a voice jumping from one stream to another).
  3. Fine-tuning: Final training on real-world meeting data (the AMI corpus).

Datasets

  • S0 (Pre-training): A massive internal dataset of 75,000 hours of anonymized monaural English utterances.
  • AMI Meeting Corpus: The primary benchmark for conversational speech. It contains ~100 hours of meetings recorded with an 8-channel microphone array. The authors used the standard development and evaluation splits.

Key Insights and Ideas

  • Fixing the Front-End in the Back-End: The most clever part of this paper is acknowledging that speech separation isn't perfect. By intentionally simulating "leakage" and "splits" during training, they taught the ASR model to ignore the artifacts and errors produced by the VarArray front-end.
  • Edge-to-Cloud Efficiency: Because VarArray can run on a local device (the "edge") and reduce 8+ microphone channels down to just two audio streams, it significantly lowers the data transmission costs to the ASR server.
  • Hardware Independence: Because the system is geometry-agnostic, a company could deploy this software across different meeting rooms with different hardware setups without needing to retrain the model for each room.

Key Results

The t-SOT-VA system achieved State-of-the-Art (SOTA) results on the AMI corpus:

  • Word Error Rate (WER): 13.7% on the development set and 15.5% on the evaluation set.
  • Comparison: This significantly outperforms previous benchmarks (which were often in the 16-20%+ range) while maintaining a total algorithmic latency of only 0.8 seconds.
  • Scaling: The system proved robust; adding more microphones (from 1 to 8) consistently improved performance without requiring a model restart or retraining.

Limitations and Future Work

  • Speaker Limit: The current implementation is optimized for $M=2$ (two people speaking simultaneously). While it can handle a meeting with many people, if three or more people talk at the exact same millisecond, the system may struggle.
  • The "Distant" Gap: Even with SOTA results, there is still a performance gap. The WER on distant microphones (15.5%) is still higher than the WER using individual close-talk headsets (9.9%). Closing this "distant-to-close" gap remains the ultimate goal for the field.
  • Computational Cost: The larger version of the model (TT36) has 142 million parameters, which might be heavy for some real-time edge deployments, though the authors did test a smaller 2 million parameter VarArray model to show feasibility.

Summary updated

Abstract

This paper presents a novel streaming automatic speech recognition (ASR) framework for multi-talker overlapping speech captured by a distant microphone array with an arbitrary geometry. Our framework, named t-SOT-VA, capitalizes on independently developed two recent technologies; array-geometry-agnostic continuous speech separation, or VarArray, and streaming multi-talker ASR based on token-level serialized output training (t-SOT). To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated speech signals from VarArray. We also propose a pre-training scheme for such an ASR model where we simulate VarArray’s output signals based on monaural single-talker ASR training data. Conversation transcription experiments using the AMI meeting corpus show that the system based on the proposed framework significantly outperforms conventional ones. Our system achieves the state-of-the-art word error rates of 13.7% and 15.5% for the AMI development and evaluation sets, respectively, in the multiple-distant-microphone setting while retaining the streaming inference capability.
BibTeX
@inproceedings{kanda2023vararray,
  author = {Naoyuki Kanda and Jian Wu and Xiaofei Wang and Zhuo Chen and Jinyu Li and Takuya Yoshioka},
  title = {{Vararray Meets T-Sot: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095367},
}
Figure