In a typical conversation, people don't wait for a perfect silence to start speaking. They interrupt, laugh, and talk over one another. For decades, Automatic Speech Recognition (ASR) systems struggled with this "cocktail party problem," usually attempting to solve it by first physically separating the audio into clean, individual streams before transcribing them.
A new paradigm, pioneered by researchers like Hiroshi Seki and Shinji Watanabe, moves away from this "separate-then-recognize" approach. Instead, it treats multi-speaker transcription as a single sequence-to-sequence problem. This direction, known as Serialized Output Training (SOT), allows a single model to transcribe overlapping voices by linearizing multiple speaker streams into one continuous sequence of tokens.
The Intuition of Linearization
The core breakthrough of SOT is the realization that we don't need separate output "heads" or complex audio unmixing to handle multiple people. Instead, we can represent a multi-party conversation as a single string of text where special symbols act as "traffic lights" for the decoder.
In the first purely end-to-end system for multi-speaker ASR, Seki et al. demonstrated that a model could learn to output the transcript of Speaker A, followed by a special "speaker-change" token, and then the transcript of Speaker B—all from a single mixed audio input. This approach, further refined in the M2MeT challenge by Fan Yu and Shiliang Zhang, avoids the "permutation problem"—the mathematical headache of trying to decide which output branch should correspond to which speaker.
From Utterances to Tokens
While early SOT models transcribed one speaker's entire sentence before moving to the next, this created high latency. If Speaker A talked for a minute while Speaker B made a quick comment, the system wouldn't show Speaker B's words until Speaker A finished.
To solve this, Naoyuki Kanda and colleagues at Microsoft introduced token-level Serialized Output Training (t-SOT). In this framework, the model interleaves words from different speakers in the exact chronological order they are spoken.
Imagine a transcript that looks like this:
[Spk1] How [Spk2] I'm [Spk1] are [Spk2] fine [Spk1] you?
By inserting these speaker-transition tokens at the word level, Kanda et al. enabled streaming ASR that can attribute speech to specific individuals in real-time. This method effectively collapses the computational complexity of multi-talker recognition from cubic (in traditional permutation-based methods) to linear, as shown in work by Yu Wu on large-scale pre-training.
Maintaining Conversational Threads
A significant challenge of interleaving tokens is that it breaks the linguistic flow. A standard Language Model (LM) expects "How are you?" to be a continuous sequence; it gets confused when "I'm fine" is wedged in the middle.
To fix this, Jian Wu and Naoyuki Kanda developed a "state-switching" mechanism in the t-SOT FNT architecture. The model maintains independent hidden states for each concurrent speaker. When the model encounters a speaker-change token, it "swaps" the active memory of the LM. This allows the model to treat the interleaved stream as two (or more) coherent, parallel sentences, significantly improving accuracy in overlapping segments.
Unifying "What" and "Who"
The SOT framework has proven flexible enough to absorb other tasks that were previously handled by separate, brittle modules:
- Diarization: Instead of a separate clustering step to figure out "who spoke when," models like SLIDAR by Samuele Cornell and Shinji Watanabe use serialized output to perform joint diarization and recognition.
- Speaker Roles: Anfeng Xu demonstrated that SOT can be used to tag speaker roles (e.g., "Doctor" vs. "Patient") directly in the transcript.
- Translation: Systems like DiariST by Mu Yang and Naoyuki Kanda apply t-SOT to speech translation, allowing a single model to translate overlapping speakers into a target language while maintaining speaker attribution.
Why It Works: The Power of Context
The reason SOT often outperforms modular systems is contextual synergy. In a modular system, the diarization module only looks at acoustic features (the sound of the voice) to decide who is speaking. In an SOT system, the model can use linguistic context. If the transcript says "How are...", the model's language component strongly expects the next word to be "you" from the same speaker, helping it resolve acoustic ambiguities during heavy overlap.
By treating the "cocktail party" as a sequence modeling problem, researchers have moved from trying to "fix" the audio to simply "reading" the conversation as it naturally unfolds.