In the pursuit of isolating specific sounds—whether it is a target speaker’s voice, a dog barking, or a specific spatial "cone" of audio—researchers have encountered a persistent failure mode: hallucination. When a model is asked to extract a signal that isn't actually there, it often produces distorted artifacts, leaks background noise, or incorrectly isolates a non-target source.
To combat this, a shared research direction has emerged across speech recognition, source separation, and audio-visual localization. The core idea is simple but transformative: explicitly training models to output absolute silence when a target is absent.
The Problem of the "Always-Present" Assumption
Traditional training regimes for extraction tasks typically present the model with a mixture and a corresponding target signal. This creates a dangerous inductive bias: the model learns that for every query, there must be an answer.
As noted by Keisuke Kinoshita, Tomohiro Nakatani, and Tsubasa Ochiai in Listen only to me! How well can target speech extraction handle false alarms?, real-world conditions rarely match this assumption. In a conversation, a target speaker may stop talking, or a user might query a sound category (like "siren") that isn't present in the current environment. Without a principled way to handle absence, models "hallucinate" speech or noise to satisfy the query.
Mechanism: Training for Silence
The solution involves a technique often referred to as negative sampling or silence augmentation. Researchers introduce training samples where the target source is entirely missing from the input mixture. The model is then supervised with a "zero-signal" ground truth—essentially a tensor of absolute silence.
- Speaker Extraction: In X-TaSNet, Bingsheng He and colleagues use a loss function that mandates near-silent output when the target speaker is absent. Similarly, Tongtao Ling in MC-LExt applies a logarithmic mean squared error loss to encourage absolute energy suppression in negative cases.
- Universal Sound Selection: Keisuke Kinoshita and the NTT team demonstrated in Listen to What You Want that this approach prevents models from incorrectly mapping background noise to a target category when the query is not satisfied.
- Audio-Visual Segmentation: To prevent models from segmenting objects just because they look like they should be making noise, Jinlan Li and others in Do Audio-Visual Segmentation Models Truly Segment Sounding Objects? assign empty ground-truth masks to samples with irrelevant audio, forcing the model to "listen" before it segments.
Mitigating Hallucinations in Large Models
This concept extends beyond signal extraction into generative speech models. Large-scale models like OpenAI's Whisper often hallucinate text when encountering non-speech sounds like wind or music.
In Calm-Whisper, researchers identified that specific attention heads are responsible for these "crazy" hallucinations. By fine-tuning these components on pure noise paired with empty text labels, they reduced hallucination rates by over 80%. Similarly, Jun Zhang and colleagues in A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASR found that pairing non-speech audio with empty transcripts effectively mitigates insertion errors in noisy environments.
The Trade-off: Over-Suppression
While negative sampling makes models more robust, it introduces a new challenge: over-suppression. If a model becomes too sensitive to the "silence" signal, it may incorrectly mute legitimate speech segments, especially in low-resource or high-noise environments.
As Marvin Borsdorf and Haizhou Li suggest in Universal Speaker Extraction, the field is moving toward integrated classification mechanisms—systems that explicitly decide whether a speaker is present before attempting extraction—to balance the need for silence with the requirement for high-fidelity signal recovery.