The Volcspeech System for the ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge

Jun Zhang, Chen Shen, Yi Liu, Wenzhi Fan, Bin Wang, Shixue Wen, Yao Tian, Jun Zhang 0066, Jingsheng Yang, Zejun MaICASSP 2022 · DOI

TL;DR

This paper presents the Volcspeech system for the ICASSP 2022 Multi-channel Multi-party Meeting Transcription (M2MeT) Challenge. The system tackles two tracks: speaker diarization and multi-speaker ASR, achieving state-of-the-art results by combining advanced front-end processing, novel clustering techniques, and an end-to-end Conformer-based ASR architecture with serialized output training.

What Problem Does This Paper Address?

The paper addresses the significant challenges in transcribing multi-channel, multi-party meetings. These environments are acoustically complex, characterized by reverberation, overlapping speech from multiple speakers, and varying acoustic conditions. Specifically, the authors tackle:

  • Speaker Diarization: Accurately identifying "who spoke when" in noisy, overlapping speech is difficult. Traditional clustering methods struggle with high overlap rates and can miss speakers.
  • Multi-Speaker Automatic Speech Recognition (ASR): Transcribing speech when multiple people talk simultaneously is a long-standing challenge. Existing ASR systems often perform poorly in such scenarios, leading to significant errors.

The M2MeT challenge aims to push the boundaries of these technologies in realistic meeting scenarios.

Methods

The Volcspeech system is divided into two main tracks:

Track 1: Speaker Diarization

The system builds upon a clustering-based approach, enhancing it with several novel components:

  • Front-end Processing:
    • Dereverberation: A multi-channel Kalman filtering algorithm is used to reduce reverberation, a common issue in meeting rooms.
  • Direction-of-Arrival (DOA) Estimation: A neural network estimates the direction of sound sources, providing valuable spatial information.
  • Speaker Embedding: Two architectures, ResNet-101 and ECAPA-TDNN, are used to extract speaker embeddings. Data augmentation techniques like speed perturbation and RIRs are employed.
  • Clustering: The core clustering is done using VBx and spectral clustering. The authors propose a modified DOVER-Lap for system fusion, which improves the estimation of the number of speakers in overlapped regions.
  • Overlap Detection: Dedicated models are trained to identify segments where speech overlaps.
  • Multi-channel Combination: Results from individual channels are combined to reduce missed speaker errors, leveraging the fact that speakers might be clearer in different channels.
  • DOA Integration: The DOA information is integrated into the VBx framework, using state-specific distributions and a novel transition probability model based on DOA changes.

Track 2: Multi-Speaker ASR

The ASR system is an end-to-end (E2E) model with a focus on handling multi-speaker and overlapping speech:

  • E2E ASR Structure: A joint CTC-attention Conformer-based encoder-decoder model is used. This architecture combines the strengths of CTC for alignment and attention for sequence modeling.
  • Serialized Output Training (SOT): This technique is crucial for multi-speaker ASR. It serializes multiple speaker transcriptions into a single sequence, allowing the model to learn dependencies between speakers.
  • Neural Front-end Speech Enhancement: An E2E multi-channel module is integrated to process the 8-channel audio. This module uses complex convolutions and self-attention to exploit spectral and spatial information, enhancing the input for the ASR back-end.
  • Data Augmentation: Extensive data augmentation is performed, including simulating reverberation, additive noise, and varying speaker overlap, to improve robustness.
  • Language Model Fusion: N-gram and transformer LMs are fused with the ASR output to further improve performance.

Datasets

  • Speaker Diarization:
    • Training: CN-Celeb, AliMeeting Corpus (simulated and real), AISHELL-4.
  • Evaluation: AliMeeting Eval and Test sets.
  • ASR:
    • Training: AliMeeting (far-field and near-field), AISHELL-4, and a large simulated dataset (600 hours) created by convolving real audio with Room Impulse Responses (RIRs), additive noise (MUSAN, Freesound), and varying speaker overlap.
  • Evaluation: AliMeeting Eval and Test sets.

Key Insights and Ideas

  • Multi-channel information is key: Leveraging audio from multiple microphones significantly improves both speaker diarization and ASR, especially in challenging overlapping speech scenarios.
  • DOA estimation aids diarization: Incorporating direction-of-arrival information into speaker diarization models provides valuable cues for distinguishing speakers and handling overlaps.
  • Serialized Output Training (SOT) is effective for multi-speaker ASR: This method allows the model to learn the sequential nature of speech in multi-party conversations, leading to better transcription accuracy.
  • End-to-end neural front-ends are powerful: Integrating speech enhancement directly into the ASR pipeline, rather than as a separate pre-processing step, allows for joint optimization and better adaptation to acoustic conditions.
  • Data augmentation is critical for robustness: The extensive use of simulated data and augmentation techniques is essential for training robust models that can generalize to unseen meeting conditions.

Key Results

Speaker Diarization (Track 1): * The final submission achieved a DER of 5.79% on the Eval set and 7.23% on the Test set. This represents a significant improvement over the baseline systems.

ASR (Track 2): * The system achieved a CER of 19.2% on the Eval set and 20.8% on the Test set. * The neural front-end significantly improved performance on overlapping utterances. * Data augmentation, particularly the simulated dataset, led to substantial CER reductions. * Model averaging and transformer LM fusion further boosted performance.

Limitations and Future Work

  • Speaker Diarization: While the system achieved excellent results, the paper mentions that more details on integrating azimuth information will be released in the future. The effectiveness of DOA integration could be further explored.
  • ASR: The paper highlights the effectiveness of the neural front-end for overlapping speech. Future work could explore more advanced neural front-end architectures or different integration strategies. The paper also mentions that the system is based on a Conformer architecture, and further exploration of other advanced architectures could be beneficial. The authors also suggest that further improvements could be achieved by exploring different LM fusion strategies.

Summary updated

BibTeX
@inproceedings{zhang2022volcspeech,
  author = {Jun Zhang and Chen Shen and Yi Liu and Wenzhi Fan and Bin Wang and Shixue Wen and Yao Tian and Jun Zhang and Jingsheng Yang and Zejun Ma},
  title = {{The Volcspeech System for the ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747381},
}
Figure