The traditional approach to the "cocktail party problem"—transcribing audio where multiple people speak at the same time—relies on a two-stage pipeline. First, a source separation module untangles the overlapping audio into clean, isolated signals. Second, an Automatic Speech Recognition (ASR) model transcribes each isolated stream independently.
While intuitive, this modular approach has severe drawbacks. Source separation algorithms are computationally expensive, introduce processing artifacts, and optimize for intermediate acoustic metrics rather than final transcription accuracy. Alternatively, end-to-end multi-branch networks attempt to output multiple text streams in parallel, but they scale poorly, require a fixed maximum number of speakers, and suffer from permutation ambiguity during training.
Serialized Output Training (SOT) bypasses these complexities by reframing multi-talker speech recognition as a single-sequence generation task. Instead of separating the audio signals or branching the network architecture, SOT collapses multiple overlapping speaker transcripts into a single, continuous token sequence. By training a standard sequence-to-sequence or transducer model on these serialized targets, the network implicitly learns to disentangle the acoustic mixture and track speaker transitions entirely within its internal representations.
Sentence-Level Serialization
The foundational SOT framework, introduced by Naoyuki Kanda et al. in Serialized Output Training for End-to-End Overlapped Speech Recognition, serializes multi-speaker transcripts at the sentence level. When two or more speakers overlap, their full utterances are concatenated one after the other into a single target string, separated by a special delimiter token (such as <sc> for speaker change).
Acoustic Mixture: [Speaker 1: "How are you?"]
[Speaker 2: "I'm doing well!"]
Serialized Target: "How are you? <sc> I'm doing well! <eos>"
This simple formatting shift completely changes how the model processes overlapping speech:
- Shared Contextual Awareness: By generating transcripts sequentially through a shared decoder, the model maintains a continuous history of what it has already transcribed. This explicit textual context prevents the model from repeating words or transcribing the same speaker twice—a frequent failure mode in parallel multi-branch systems.
- Dynamic Speaker Counting: The model naturally determines the number of speakers in the audio mixture. It continues decoding text through the speaker-change tokens until it emits a final end-of-sequence (
<eos>) token, removing the need for a hardcoded maximum speaker limit. - Zero Architectural Overhead: Because the output layer remains a single branch, a model trained with SOT can seamlessly transition between single-speaker and multi-speaker audio without requiring architectural modifications or sacrificing performance on clean, non-overlapping speech.
Chronological Interleaving for Streaming Latency
While sentence-level SOT is highly effective for offline transcription, it introduces a fundamental latency bottleneck. The model cannot begin outputting the second speaker's words until the first speaker has completely finished talking. For real-time applications like live captioning, this delay is unacceptable.
To enable streaming multi-talker ASR, the author, Jian Wu, proposed token-level Serialized Output Training (t-SOT) in Streaming Multi-Talker Speech Recognition with Token-Level Serialized Output Training. Instead of waiting for an entire sentence to conclude, t-SOT interleaves tokens from different speakers in strict chronological order based on their exact emission times in the audio stream.
Acoustic Mixture: [Speaker 1: "How are you?"]
[Speaker 2: "I'm doing well!"]
Serialized Target: "How <cc> I'm <cc> are doing <cc> you? well! <eos>"
Whenever the active speaker changes in the timeline, the model emits a special "channel-change" token (<cc>). This fine-grained interleaving allows a single-branch architecture, such as a Transformer Transducer, to output partial results for multiple concurrent speakers simultaneously with minimal latency.
However, interleaving words from different conversations breaks the linguistic structure of the text, which can confuse the model's internal language modeling capabilities. To resolve this, advanced t-SOT architectures decouple the linguistic context during inference. As detailed by Jian Wu et al. in t-SOT FNT: Streaming Multi-talker ASR with Text-only Domain Adaptation Capability, the system maintains distinct hidden states for each virtual speaker channel. When a channel-change token is predicted, the model switches its internal language model state:
[Predicting Speaker 1] -> State 1 updated -> Emits <cc> -> Switch to State 2 -> [Predicting Speaker 2]
This state-switching mechanism ensures that the language model processes each speaker's output as a continuous, grammatically coherent sentence, closing the performance gap between multi-talker and specialized single-speaker models.
Joint Transcription and Diarization
The intuition behind serialization extends beyond text; it can also solve the "who spoke when" problem (speaker diarization). By treating speaker identities, transitions, and attributes as tokens within a unified vocabulary, sequence transduction models can perform transcription and speaker attribution simultaneously.
In Joint Speech Recognition and Speaker Diarization via Sequence Transduction, Laurent El Shafey et al. demonstrated that embedding speaker labels directly into the output transcript allows the network to leverage both acoustic features and conversational context. For example, the model can learn that a question from "Speaker A" is linguistically likely to be followed by a response from "Speaker B," using this context to sharpen its speaker boundary predictions.
Similarly, Naoyuki Kanda et al. in Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of any Number of Speakers integrated an external "speaker profile inventory" directly into the serialized decoding process. The model autoregressively generates a single sequence containing text tokens interleaved with speaker identification tags, dynamically mapping the overlapping audio to specific voice profiles in a single pass.
The Generative Flip Side: Full-Duplex Dialogue
The mechanism of linearizing parallel audio streams into a single interleaved sequence is equally powerful in reverse. Recent breakthroughs in speech generation apply this exact intuition to synthesize realistic, multi-talker human interactions.
Traditional spoken dialogue systems operate half-duplex: the AI waits for the user to finish speaking before generating a response. To achieve natural, full-duplex conversations—complete with interruptions, overlapping laughter, and backchannel feedback (like "mhm" or "yeah")—frameworks like CoVoMix (Lei He et al.) and Behavior-SD (Sehun Lee et al.) utilize single-stream sequence representations.
By training high-capacity Transformer architectures on text and audio tokens interleaved with specialized control symbols for gaps and overlaps, the attention mechanism naturally learns the temporal and prosodic relationships between speakers. Instead of managing complex parallel audio generation tracks, the model treats a multi-party dialogue as a single, serialized stream, capturing the messy, synchronous flow of human conversation.
Key Foundational Work
Speech interaction strategies for a humanoid assistant (2018) — contributed 2 concept elements to this direction.