In a "cocktail party" scenario where multiple people speak simultaneously, a neural network faces a fundamental mathematical hurdle known as label ambiguity. If a model is trained to separate two voices into two output channels, how does it know which speaker belongs in which channel? Without a consistent rule, the model receives conflicting gradients—being told to put Speaker A in Channel 1 in one instance and Channel 2 in another—which prevents it from ever converging.
The Foundation: Permutation Invariant Training
The breakthrough solution to this problem was Permutation Invariant Training (PIT), introduced by Yanmin Qian and Dong Yu in Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation.
The intuition behind PIT is simple: instead of forcing a fixed assignment, the system calculates the loss for every possible permutation of speakers-to-outputs. It then backpropagates only the minimum loss. This allows the network to "choose" the most natural assignment for any given mixture, effectively letting the model organize its own output streams. This approach was later extended to Single-Channel Multi-talker Speech Recognition, proving that the same logic could solve "who spoke what" in transcription tasks.
The Factorial Bottleneck
While PIT works elegantly for two or three speakers, it faces a severe scaling problem. As noted in the Asteroid toolkit and by researchers like Marc Delcroix in End-to-End SpeakerBeam, the number of possible permutations grows factorially ($O(C!)$) with the number of speakers $C$.
For a 10-speaker mixture, there are over 3.6 million possible assignments. Evaluating all of them to find the minimum loss becomes computationally impossible. This "combinatorial explosion" has historically limited end-to-end models to handling only a few concurrent voices.
Scaling to Many Speakers
To break the factorial bottleneck, researchers have moved toward more efficient assignment algorithms and architectural shifts:
- The Hungarian Algorithm: Shaked Dovrat and colleagues demonstrated in Many-Speakers Single Channel Speech Separation with Optimal Permutation Training that the permutation problem can be framed as a bipartite matching problem. By using the Hungarian algorithm, they reduced the complexity from $O(C!)$ to $O(C^3)$, enabling the separation of up to 20 concurrent speakers.
- Serialized Output Training (SOT): Instead of parallel output channels, Naoyuki Kanda and the Microsoft team proposed Serialized Output Training. SOT collapses multiple speaker transcripts into a single sequence separated by special tokens. This bypasses permutations entirely by forcing a specific order (usually based on start times), allowing models to scale to large-scale meeting transcriptions.
- Sort Loss: In Sortformer, Taejin Park introduced a "Sort Loss" that resolves permutations by ordering speakers based on their arrival time. This provides a deterministic supervision signal that eliminates the need for exhaustive searching during training.
Handling Unknown Speaker Counts
A secondary challenge is that the number of speakers in a real-world recording is rarely known in advance. Traditional PIT requires a fixed number of output layers.
To solve this, Naoya Takahashi proposed Recursive Speech Separation, which uses a "One-and-Rest" approach to peel away speakers one by one until only noise remains. Alternatively, Shota Horiguchi developed Encoder-Decoder Based Attractors (EDA) in End-to-End Speaker Diarization. EDA generates a flexible number of "attractor" vectors that represent individual speakers, allowing the model to adapt its output to however many people are actually talking.
From Permutations to Streams
Recent work has shifted toward making these systems practical for real-time use. Liang Lu introduced the Streaming Unmixing and Recognition Transducer (SURT), which applies heuristic-based losses to minimize latency. To further improve efficiency, Xiluo He and others have explored Speaker-Agnostic Activity Streams, which collapse many speakers into just two streams. This reduces the computational load of inference, which otherwise scales linearly with the number of participants, making high-density conversation tracking feasible on standard hardware.