Imagine being at a loud cocktail party with five people speaking simultaneously, but you only have a single microphone recording the room. Your goal is to isolate each individual voice.
Mathematically, we can express the observed mixture $x(t)$ as a combination of $N$ independent source signals $s_1(t), s_2(t), \dots, s_N(t)$ mixed through some matrix $A$:
$$x(t) = A s(t)$$
If we have at least as many microphones as speakers, the problem is determined or overdetermined, and we can invert the mixing matrix using standard linear techniques like Independent Component Analysis (ICA). But when the number of sources exceeds the number of sensors—or in the extreme case of a single microphone—the mixing matrix $A$ becomes non-invertible.
This is the core challenge of underdetermined source separation. Because there are more unknowns than equations, an infinite number of source combinations can sum to the exact same observed mixture. As noted by Aarthi M. Reddy in the context of single-channel speaker separation, the problem is fundamentally underconstrained. To resolve this ambiguity and uniquely recover the original signals, we must look beyond pure linear algebra and enforce structural, statistical, or generative priors.
Exploiting Sparsity in the Time-Frequency Domain
The most powerful escape hatch from the underdetermined deadlock relies on a simple observation: while multiple audio sources overlap heavily in the time domain, they rarely overlap completely in the time-frequency (TF) domain.
By transforming a single-channel or multi-channel mixture using the Short-Time Fourier Transform (STFT), we can exploit a property known as W-disjoint orthogonality. In this sparse representation, each specific time-frequency bin is typically dominated by just a single active source. This insight transforms an intractable continuous unmixing problem into a discrete assignment problem: for every point in the spectrogram, which source does it belong to?
Afsaneh Asaei et al. demonstrate the efficacy of this approach in Sparse component analysis for speech recognition in multi-speaker environment, showing that Sparse Component Analysis (SCA) can effectively isolate overlapping talkers by leveraging domain sparsity.
To determine these assignments blindly, algorithms cluster the data points based on shared features. For instance, the authors of MAP-Based Underdetermined Blind Source Separation of Convolutive Mixtures by Hierarchical Clustering and ℓ1-Norm Minimization combine hierarchical clustering with $\ell_1$-norm minimization to group and reconstruct sparse components. Even when a small number of spatial sensors are available, clustering remains vital. Michael I. Mandel et al. utilize an Expectation-Maximization (EM) algorithm in An EM Algorithm for Localizing Multiple Sound Sources in Reverberant Environments to analyze interaural phase differences across two microphones, probabilistically masking and assigning TF bins to their respective spatial origins.
Constraining Solutions via Matrix and Tensor Factorization
Sparsity alone can falter when sources do overlap significantly or when room reverberation smears signals across time and frequency. To handle these complex mixtures, researchers enforce global structural constraints on the underlying source spectrograms.
A key intuition here is that real-world sounds are highly redundant. A spoken word or a musical instrument consists of a small set of recurring spectral patterns (templates) activated over time. Non-negative Matrix Factorization (NMF) exploits this by decomposing a mixed spectrogram into a template matrix and an activation matrix, ensuring all values remain non-negative.
Jonathan Le Roux et al. expand on this in Adaptive Template Matching with Shift-Invariant Semi-NMF, combining shift-invariance with sparsity constraints to automatically discover, cluster, and time-align overlapping events from a single channel.
From a geometric perspective, framing source separation as a convex optimization problem allows researchers to establish provable boundaries on when a mixture can be successfully decomposed. As detailed by Michael B. McCoy et al. in Convexity in Source Separation : Models, geometry, and algorithms, low-dimensional structural priors can reliably guide optimization algorithms away from trivial solutions. For example, Augustin Lefèvre et al. introduce a nuclear-norm based convex formulation in A nuclear-norm based convex formulation for informed source separation to encourage low-rank structures, successfully bypassing the local minima that frequently plague non-convex optimization techniques.
Deep Generative Priors and Temporal Context
When spatial cues are entirely absent—such as separating a singing voice from background instruments in a monaural track—the system must rely on explicit models of what the target sources are supposed to sound like.
Early frameworks achieved this by modeling the temporal dynamics of signals. For example, the authors of Maximum Likelihood Blind Source Separation: A Context-Sensitive Generalization of ICA utilize autoregressive models to capture temporal context, while the framework in Variational Speech Separation of More Sources than Mixtures employs source-specific Hidden Markov Models (HMMs) to provide the statistical constraints necessary to isolate individual voices.
Modern architectures replace these hand-crafted statistical models with deep neural networks capable of learning high-dimensional generative distributions. Po-Sen Huang et al. pioneer this in Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation by training Deep Recurrent Neural Networks (DRNNs) alongside a time-frequency masking layer. Crucially, they introduce a reconstruction constraint—forcing the separated outputs to sum perfectly back to the original mixture—which drastically reduces search space ambiguity.
┌───────────────┐
│ Mixed Audio │
│ Signal x(t) │
└───────┬───────┘
│
▼
┌───────────────────────────┐
│ Deep Generative Prior │
│ (e.g., Diffusion / DRNN) │
└─────────────┬─────────────┘
│
┌─────────────┴─────────────┐
│ Reconstruction Constraint│
│ s_1(t) + s_2(t) = x(t)│
└─────────────┬─────────────┘
│
┌─────────────────┴─────────────────┐
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Source 1 │ │ Source 2 │
│ (Isolated) │ │ (Isolated) │
└───────────────┘ └───────────────┘
More recently, diffusion models have emerged as an exceptionally robust prior for solving these ill-posed inverse problems. In BUDDy: Single-Channel Blind Unsupervised Dereverberation with Diffusion Models, Eloi Moliner et al. pair a generative diffusion model trained on clean speech with a physics-informed model of room acoustics. Because the diffusion model acts as a strict mathematical anchor for what constitutes "clean speech," the system can blindly untangle room reflections from the underlying voice using only a single channel, demonstrating how strong generative models can elegantly resolve the underdetermined bottleneck.