In a crowded room, the human auditory system performs a remarkable feat: it can "lock onto" a single conversation while treating a dozen others as background noise. For decades, researchers have attempted to replicate this "cocktail party" effect in machines through Target Speaker Extraction (TSE).
The traditional approach to TSE relies on a "voiceprint"—a pre-recorded, clean audio sample of the target speaker used to "enroll" them in the system. However, as noted by Desh Raj and Leibny Paola García-Perera, this requirement is a major bottleneck. In spontaneous real-world scenarios—like a forensic investigation, a live presentation, or a meeting with unannounced guests—a clean reference recording simply does not exist.
Recent research has shifted toward a more flexible paradigm: using multimodal "anchors" or prompts to define the target speaker on the fly, bypassing the need for prior enrollment.
The Anchor Problem
The fundamental challenge of speaker extraction is permutation ambiguity: if a model separates two voices, how does it know which one to give to the user? A voiceprint solves this by providing a template to match. Without it, the system needs a different way to "query" the audio mixture.
Researchers are now using a variety of non-auditory cues to serve as these queries:
- Visual Anchors: Zexu Pan and Haizhou Li have demonstrated that the temporal synchronization between lip movements and speech can act as a powerful filter. In their MuSE and reentry models, the system "self-enrolls" by using the visual rhythm of the speaker's mouth to identify which acoustic features in the mixture belong to that person. Similarly, Soyeon Choe and Hong-Goo Kang developed FaceFilter, which uses a single still image of a face to extract a voice, leveraging the biological correlation between facial structure and vocal tract characteristics.
- Textual and Semantic Anchors: Instead of a voice sample, a user might provide a description. Jibin Wu and Kay Chen Tan introduced LLM-TSE, which allows users to "type to listen" using natural language commands. In specific contexts like academic talks, Jiahe Lei and Wei Xue showed that keywords and bullet points from presentation slides (pTSE-T) can guide the extraction of the presenter's voice by matching the audio content to the text on screen.
- Spatial and Neural Anchors: Physical location can also serve as a prompt. Dan Su and Rongzhi Gu developed a Neural Spatial Filter that uses a speaker's direction as an anchor. More speculatively, Zexu Pan and Tanja Schultz have explored NeuroHeed, which uses EEG signals to detect which speaker a human is focusing on, using brain activity as the extraction query.
From Identity to Attributes
This shift represents a move from identity-based extraction (who is speaking?) to attribute-based extraction (what are they doing/where are they?).
The framework proposed by Yidi Jiang and Haizhou Li, known as MM-TSD, exemplifies this. It treats speaker extraction as a retrieval task where the prompt can be an image, a text description ("find the male speaker"), or even a logical exclusion ("everyone except this person").
The intuition behind why this works lies in cross-modal embedding spaces. By training models on massive datasets of aligned audio, video, and text, researchers create a shared mathematical space where a "face" vector and a "voice" vector for the same person are close together. When a user provides a still image or a text prompt, the model projects that prompt into the shared space and uses it as a "key" to attend to the corresponding "values" in the audio mixture.
Self-Enrollment and Blind Extraction
When no external prompt is available, the system must look inward. Cong Han 0001 and Shinji Watanabe developed CSSUSI, which uses K-means clustering to build a "speaker inventory" directly from a long recording. By identifying recurring vocal patterns within the mixture itself, the model can separate speakers without any prior knowledge of their identities.
This trajectory suggests a future where "selective hearing" devices do not need to be pre-programmed with our friends' voices. Instead, they will use the context of our environment—the person we are looking at, the slides we are reading, or the direction we are facing—to decide which voice matters most in the moment.