Balancing Perceptual Quality and Signal Fidelity in Generative Audio

21 concepts11 papers1 proposal

When evaluating an audio restoration model, it seems intuitive that an output that sounds cleaner, crisper, and more natural to a human listener should be considered objectively better. However, in modern generative audio modeling, the opposite is frequently true: models that achieve the highest subjective scores from human listeners often perform poorly on traditional signal reconstruction metrics.

This tension defines a fundamental conflict in generative audio processing. To understand why this happens, we have to look at how generative models reconstruct sound compared to traditional discriminative approaches, and why the very mechanism that makes audio sound good to humans causes it to fail standard mathematical benchmarks.


The Two Definitions of Quality

Historically, speech enhancement and audio restoration tasks were dominated by discriminative models. As noted by Philippe Gonzalez et al. in their work on diffusion-based speech enhancement, discriminative models prioritize exact signal fidelity. They are typically trained using point-wise regression losses (like Mean Squared Error or $L_1$ loss) to mathematically subtract noise from a corrupted signal.

While this approach yields high signal-to-noise ratios (SNR) and high signal-to-distortion ratios (SDR), it often produces audio that sounds muffled, robotic, or unnatural to human ears. Because the model is forced to minimize the average error across all possible clean signals, it plays it safe by smoothing out the fine-grained details.

Discriminative Approach (Point-wise Error Minimization):
Noisy Input ───► [Minimize Distance to Target] ───► Muffled, mathematically close output

Generative Approach (Distribution Matching):
Noisy Input ───► [Sample from Clean Distribution] ───► Crisp, natural, but slightly shifted output

Generative models—such as those based on diffusion, Schrödinger bridges, or discrete neural codecs—shift the goal from minimizing distance to matching distributions. Instead of filtering the noise away, they treat restoration as a conditional generation task: given a noisy or degraded input, what is a highly plausible clean audio signal that matches this context?

As a result, generative models excel at producing audio with high human-perceived quality and intelligibility. However, as demonstrated by Julius Richter et al. in a rigorous comparison of generative frameworks, maximizing these perceptual scores almost invariably leads to a drop in traditional signal-to-distortion metrics.


Why Generative Re-Synthesis Penalizes Fidelity

To understand why high perceptual quality tanks signal fidelity metrics, consider how metrics like Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) operate. These metrics perform a sample-by-sample, point-wise comparison between the reconstructed waveform and the original ground-truth clean waveform. They require perfect alignment in both time and phase.

Generative models do not perform direct subtraction; they re-imagine or re-synthesize missing or corrupted components of the audio signal. In their study on generative refinement, Naoki Murata et al. explain that because these models re-synthesize audio components rather than strictly preserving the original signal's phase or structure, the reconstructed waveform inevitably deviates from the original source.

Even if a diffusion model generates a perfectly clean harmonic that sounds identical to the original, a microscopic shift in phase or a tiny temporal misalignment will cause a point-wise metric to register a massive error. As Shengkui Zhao et al. point out, because the model re-imagines missing components rather than performing a direct subtraction, it lacks high scale-invariant signal-to-distortion ratios, despite sounding entirely natural to human listeners.

This phenomenon is visualized by Liang Xu et al., who show that optimizing exclusively for human-perceived clarity can lead to severe waveform misalignment and structural inaccuracies compared to the ground truth.


Hallucinations and the Limits of Realism

The freedom that allows generative models to synthesize realistic audio also introduces a significant risk: content hallucination. When a model is tasked with generating plausible details to fill in heavily degraded gaps, it may introduce artificial vocal nuances, background textures, or even alter phonetic content.

In Genhancer, a model developed by Jiaqi Su et al. that operates on discrete codec tokens, the authors note that maintaining strict fidelity to the source material remains an open challenge because stochastic modeling is inherently susceptible to modifying spoken words or introducing artificial vocal characteristics.

This challenge became highly visible during the Interspeech 2025 URGENT Speech Enhancement Challenge. Organized by Kohei Saijo et al., the challenge tested various models across multiple languages and distortions. The results highlighted a stark divide: while purely discriminative models led in objective accuracy metrics, generative models were overwhelmingly preferred by human listeners. However, when faced with unfamiliar languages, the generative models frequently "hallucinated" speech, introducing phonetic artifacts that actually lowered true content fidelity and intelligibility.

This trade-off extends beyond speech enhancement into broader audio generation tasks. For instance, when generating stereo audio from mono tracks, Qiuqiang Kong et al. observed in DiffStereo that increasing the perceived spatial immersion often introduces subtle artifacts or hallucinations, causing a corresponding decrease in overall signal quality and naturalness.


Architectural and Objective Interventions

To bridge this gap, researchers are exploring specialized architectures and hybrid training objectives that force generative models to respect the underlying physical constraints of the source signal.

1. Representation Choices

The mathematical space in which a model operates heavily influences its position on the perception-fidelity spectrum. In an extension of Audio Masked Autoencoders, Zhi Zhong et al. discovered a fundamental architectural conflict: models operating on mel-spectrograms excel at human-perceived quality metrics because the mel-scale mimics human auditory perception. Conversely, Short-Time Fourier Transform (STFT)-oriented architectures achieve significantly higher raw signal fidelity relative to a clean reference signal because they preserve precise phase and linear frequency relationships.

2. Hybrid Losses

Rather than relying solely on a generative score-matching or diffusion loss, researchers are anchoring models using supervised constraints. For example, Jean-Eudes Ayilo et al. introduced a hybrid strategy that appends a weighted Mean Squared Error (MSE) loss to the standard diffusion objective. This forces the model to prioritize the reconstruction of the signal's statistical distribution while remaining structurally bound to the exact point-wise layout of the noisy input.

Hybrid Loss Function:
L_total = L_diffusion (Perceptual Naturalness) + w * L_mse (Structural Fidelity)

3. Joint Optimization and Distillation

To prevent the waveform misalignment that occurs during multi-step sampling, recent work focuses on joint optimization. The one-step speech enhancement model proposed by Liang Xu et al. uses consistency distillation to compress a complex diffusion trajectory into a single step, explicitly balancing audio transparency with strict signal accuracy to prevent temporal shifts.


Exploiting the Disconnect: Content-Aligned Watermarking

While this divergence between human perception and signal metrics is a hurdle for audio restoration, it can be uniquely exploited in fields like audio watermarking.

Traditional watermarks rely on low-energy noise that is easily wiped out by AI denoisers or neural codecs. To solve this, Haiyun Li et al. developed AlignMark, a method that embeds watermarks as a "pseudo-speech" signal matching the original audio's distribution.

Because the watermark is embedded within the speech distribution itself, it intentionally lowers raw signal-level fidelity metrics like SI-SNR and PESQ when compared to the unwatermarked source. Yet, because human perception is insensitive to these distribution-aligned shifts, subjective listening tests indicate complete imperceptibility. When a neural codec or generative denoiser processes the audio, it leaves the watermark intact because the model perceives the watermark not as noise to be stripped away, but as part of the natural speech distribution itself.

Key Foundational Work

Investigating Training Objectives for Generative Speech Enhancement (2024) — contributed 3 concept elements to this direction.
Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement (2023) — contributed 2 concept elements to this direction.
Extending Audio Masked Autoencoders Toward Audio Restoration (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_perceptual_fidelity_trade_off_c381a05d,
  title = {perceptual-fidelity trade-off},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perceptual-fidelity-trade-off/rdr_c381a05d67c54d09823e5cc5e4e2e325},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_perceptual_fidelity_trade_off_c381a05d_v1,
  title = {perceptual-fidelity trade-off},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perceptual-fidelity-trade-off/rdr_c381a05d67c54d09823e5cc5e4e2e325/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake