Cross-Speaker Encoding Network for Multi-Talker Speech Recognition

Jiawen Kang, Lingwei Meng, Mingyu Cui, Haohan Guo, Xixin Wu, Xunying Liu, Helen Meng

1. Introduction

Transcribing overlapping speech—where multiple speakers talk simultaneously—remains one of the most significant challenges in Automatic Speech Recognition (ASR). Current end-to-end solutions generally follow two paradigms: Single-Input Multiple-Output (SIMO) and Single-Input Single-Output (SISO) with Serialized Output Training (SOT).

SIMO models utilize separate branches for each speaker, but these branches operate in isolation. If one branch misinterprets a segment early on, the other branches lack a mechanism to coordinate or correct the error, often resulting in repeated or omitted words. Conversely, SOT models treat the transcript as a single continuous string. While flexible, they often struggle to "untangle" complex audio features in high-overlap scenarios because they lack explicit speaker-separation mechanisms. This paper introduces the Cross-Speaker Encoding (CSE) network, a novel architecture designed to bridge the gap between these two approaches by allowing speaker-specific branches to communicate during the encoding process.

2. Methodology

The CSE network introduces two primary innovations to address the limitations of isolated branching and serialized decoding.

2.1 The Cross-Encoder Module

In traditional SIMO models, the separation of audio into speaker-specific branches is a "one-time deal." Once split, the dependencies between speakers are lost. The CSE network solves this by introducing a Cross-Encoder. This module takes the separated speaker representations ($S_1, S_2$) and the original mixture encoding ($\hat{X}$), concatenates them, and processes them through a series of Conformer blocks.

By including the original mixture encoding ($\hat{X}$), the model retains access to the "raw" audio features, which helps recover details that might have been lost during the initial separation step. This allows the branches to effectively "talk" to one another, using the shared context to prevent the omission of words or the accidental repetition of the same word across different branches.

2.2 Partition-wise Positional Embedding (PPE)

To help the model distinguish between different speakers within the joint representation, the researchers implemented Partition-wise Positional Embedding (PPE). These are learnable embeddings that act as tags for each speaker's data partition, ensuring the Cross-Encoder can maintain speaker identity while modeling inter-speaker dependencies.

2.3 Integrated CSE-SOT Model

The authors further integrated the CSE encoder with an attention-based decoder using Serialized Output Training (SOT). In this hybrid configuration, the CSE component handles explicit feature separation and inter-speaker modeling, while the SOT decoder provides the flexibility to manage varying numbers of speakers in a single output stream. To facilitate this, they utilized a modified version of Heuristic Error Assignment Training (HEAT) that incorporates speaker-change tokens (<sc>) to explicitly mark speaker boundaries.

3. Experimental Setup

The models were evaluated using the standard LibriSpeech (LS) benchmark and LibriSpeechMix (LSM), a simulated dataset containing two-speaker (LSM-2mix) and three-speaker (LSM-3mix) scenarios. To assess the model's robustness, the researchers categorized test sets by overlap ratio:

  • Low: 0–20% overlap
  • Median: 20–50% overlap
  • High: 50–100% overlap

4. Key Results and Analysis

The CSE-SOT model demonstrated significant performance gains across all metrics compared to standard baselines.

  • Overall Accuracy: The CSE-SOT model outperformed the standard SOT baseline by 10% overall and reduced the Word Error Rate (WER) by 8% compared to the standard SIMO baseline.
  • High-Overlap Performance: The most striking improvement occurred in high-overlap scenarios (50–100% overlap), where the CSE-SOT model achieved a 16% relative improvement in WER over the SOT model. This suggests that the Cross-Encoder is particularly effective at untangling dense audio mixtures.
  • Efficiency: Despite the added architectural complexity, the CSE-SOT model maintains a parameter count of approximately 45 million, making it comparable in size to standard SOT models and suitable for practical deployment.

4.1 Attention Specialization

Visualization of the model's attention heads revealed that the network naturally learns specialized roles. Some heads focus exclusively on individual speaker cues, while others specialize in "information swapping" between branches, confirming that the model actively uses the cross-speaker communication channel to refine its transcriptions.

5. Conclusion and Future Work

The Cross-Speaker Encoding network successfully addresses the "isolation" problem in multi-talker ASR. By facilitating communication between speaker branches and retaining the original mixture context, the model significantly improves transcription accuracy in highly overlapped speech.

While the model excels in 2-speaker and 3-speaker scenarios, the authors noted a slight performance decrease when the number of speakers in the audio exceeds the number of pre-defined branches in the CSE module. Future research will focus on improving the model's generalization to an arbitrary number of speakers and optimizing the branch allocation process.

Summary updated

Abstract

End-to-end multi-talker speech recognition has garnered great interest as an effective approach to directly transcribe overlapped speech from multiple speakers. Current methods typically adopt either 1) single-input multiple-output (SIMO) models with a branched encoder, or 2) single-input single-output (SISO) models based on attention-based encoder-decoder architecture with serialized output training (SOT). In this work, we propose a Cross-Speaker Encoding (CSE) network to address the limitations of SIMO models by aggregating cross-speaker representations. Furthermore, the CSE model is integrated with SOT to leverage both the advantages of SIMO and SISO while mitigating their drawbacks. To the best of our knowledge, this work represents an early effort to integrate SIMO and SISO for multi-talker speech recognition. Experiments on the two-speaker LibrispeechMix dataset show that the CES model reduces word error rate (WER) by 8% over the SIMO baseline. The CSE-SOT model reduces WER by 10% overall and by 16% on high-overlap speech compared to the SOT model.
BibTeX
@inproceedings{kang2024cross,
  author = {Jiawen Kang and Lingwei Meng and Mingyu Cui and Haohan Guo and Xixin Wu and Xunying Liu and Helen Meng},
  title = {{Cross-Speaker Encoding Network for Multi-Talker Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=EECSSJmGy6},
}
Figure