Linguistic Fidelity in Generative Speech Restoration

This direction is split into other directions: phonetic hallucination. Its last version stays readable and citable here.

Version 1 of 176 papers109 conceptsfrozen 2026-10-04

In recent years, speech processing has undergone a paradigm shift. Traditional "predictive" models, which aim to minimize the pixel-by-pixel or sample-by-sample distance between a noisy input and a clean target, are being replaced by "generative" models—primarily diffusion-based and score-based systems. While these new models produce audio that sounds remarkably natural and "studio-quality," they introduce a deceptive new failure mode: they often hallucinate entirely new words or phonemes that were never in the original recording.

The Metric Gap: Naturalness vs. Accuracy

The central tension in modern speech restoration is the "Metric Gap." As noted in the CCF AATC 2025: Speech Restoration Challenge, there is a strong negative correlation between popular AI-based quality metrics and actual human preference.

Traditional predictive models often produce "processed" or muffled audio because they play it safe, averaging out possibilities to minimize Mean Squared Error (MSE). Generative models, such as SGMSE+ by Julius Richter and colleagues, take the opposite approach. They use a learned prior of what "clean speech" looks like to fill in missing information.

However, as Danilo de Oliveira demonstrates, this leads to a paradox: at low signal-to-noise ratios (SNR), a generative model might produce audio that sounds like a clear human voice but consists of complete "gibberish." Tal Peer argues that current "blind" quality metrics are easily fooled by these hallucinations because they prioritize acoustic texture over linguistic meaning.

Why Generative Models Hallucinate

To understand why these models hallucinate, we must look at how they balance the input signal against their learned prior.

In a diffusion model, the restoration process is a walk from a noisy state toward a clean state, guided by a "score function" that points toward the manifold of clean speech. When the input signal is severely degraded, the "pull" from the noisy evidence becomes weak. The model’s internal prior—its knowledge of how humans usually talk—takes over.

This results in several distinct types of "phonetic instability":

  1. Phonetic Drift: The model reconstructs a word that sounds plausible in context but is different from the source.
  2. Semantic Shift: Environmental noise is misinterpreted as speech, causing the model to "transcribe" background sounds into words, a phenomenon Joonyong Park found frequently in Generative Spoken Language Modeling (GSLM).
  3. Condition Collapse: The system ignores the input signal entirely, generating fluent but irrelevant speech, as discussed by Jean-Eudes Ayilo.

Anchoring Generation with Semantic Guidance

The research community is currently focused on "anchoring" these generative models to ensure they stay faithful to the original message. The most successful approach involves incorporating semantic representations from self-supervised models like HuBERT or Whisper.

For example, Xiaoyu Liu and the team at Dolby developed MaskSR2, which "distills" semantic knowledge from a pre-trained HuBERT model into the restoration process. By forcing the model to maintain the same semantic features as the original (even if noisy) signal, they reduced Word Error Rates (WER) by up to 38%.

Similarly, Jiaqi Su and Zeyu Jin introduced Genhancer, which treats audio restoration as a language modeling task. By conditioning a Transformer on both the noisy input and Whisper features, the model can "read" the intended words while "painting" the high-fidelity acoustic details.

Beyond Audio: Multi-Modal Constraints

When audio is too far gone to be recovered, researchers are turning to other "anchors." Yochai Yemini and colleagues developed LipVoicer, which uses silent video of a speaker's lips to guide the diffusion process. By predicting text from lip movements and using that text as a condition for the audio generator, the model resolves the acoustic ambiguities that lead to hallucinations in audio-only systems.

Quantifying the Problem

As these models move into production, the need for better evaluation is critical. Alkis Koudounas introduced SHALLOW, a benchmark specifically designed to decompose errors into lexical, phonetic, and semantic dimensions. This allows researchers to see when a model is "faking it"—producing high-quality audio that fails to convey the correct information.

The goal of the field is no longer just "cleaner" audio, but a synthesis of perceptual naturalness and linguistic integrity. As Marvin Sach and Yanan Fu warn in their work on the URGENT 2025 challenge, if we only optimize for how audio sounds, we risk building systems that are perfectly convincing, yet completely unreliable.

Papers in this version

Showing 60 of 76 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linguistic_fidelity_in_generative_speech_8839befc,
  title = {Linguistic Fidelity in Generative Speech Restoration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linguistic-fidelity-in-generative-speech-restoration/rdr_8839befc0e5245efb0f2d7cd5792e12a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_linguistic_fidelity_in_generative_speech_8839befc_v1,
  title = {Linguistic Fidelity in Generative Speech Restoration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linguistic-fidelity-in-generative-speech-restoration/rdr_8839befc0e5245efb0f2d7cd5792e12a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}