Spectral Collapse in Diffusion Inversion

Nicolas Bourriez, Alexandre Vérine, Auguste Genovesio

Abstract

Conditional diffusion inversion provides a powerful framework for unpaired image-to-image translation. However, we demonstrate through an extensive analysis that standard deterministic inversion (e.g. DDIM) fails when the source domain is spectrally sparse compared to the target domain (e.g., super-resolution, sketch-to-image). In these contexts, the recovered latent from the input does not follow the expected isotropic Gaussian distribution. Instead it exhibits a signal with lower frequencies, locking target sampling to oversmoothed and texture-poor generations. We term this phenomenon spectral collapse. We observe that stochastic alternatives attempting to restore the noise variance tend to break the semantic link to the input, leading to structural drift. To resolve this structure-texture trade-off, we propose Orthogonal Variance Guidance (OVG), an inference-time method that corrects the ODE dynamics to enforce the theoretical Gaussian noise magnitude within the null-space of the structural gradient. Extensive experiments on microscopy super-resolution (BBBC021) and sketch-to-image (Edges2Shoes) demonstrate that OVG effectively restores photorealistic textures while preserving structural fidelity.

Introduction: The Spectral Collapse Phenomenon

Diffusion models have become the cornerstone of modern image synthesis and editing. A common workflow for image-to-image translation involves inversion: mapping a real image back into the latent noise space. Ideally, this noise map can then be used with a new conditioning prompt to generate a version of the image that preserves the original structure while adopting new stylistic or semantic attributes.

However, a fundamental flaw emerges when the source image is "spectrally sparse"—lacking high-frequency detail, as seen in low-resolution thumbnails or line drawings. In these cases, standard deterministic inversion techniques like DDIM suffer from spectral collapse. Instead of recovering a true isotropic Gaussian noise map, the process yields a "collapsed" latent that retains a low-frequency imprint of the input.

Figure 1: Spectral collapse in standard DDIM inversion leads to structured noise maps and blurry generations, whereas OVG restores Gaussianity and high-frequency texture.

As shown in Figure 1, the spectral energy of DDIM-inverted noise drops significantly at higher frequencies. This lack of high-frequency energy forces the subsequent generation to be oversmoothed and "plastic," failing to hallucinate the necessary textures for a realistic output. While stochastic methods attempt to fix this by injecting random noise, they often cause structural drift, where the output's layout no longer matches the input.

Methodology: Orthogonal Variance Guidance (OVG)

To address the trade-off between structural fidelity and texture realism, we introduce Orthogonal Variance Guidance (OVG). OVG is an inference-time guidance strategy that modifies the inversion trajectory without requiring model retraining. It employs two primary correction terms:

  1. High-Frequency Correction ($L_{HF}$): This term penalizes deviations from the expected Gaussian noise energy, effectively restoring the missing high-frequency variance required for realistic textures.
  2. Low-Frequency Structural Correction ($L_{LF}$): This ensures the denoised estimate remains aligned with the input's global structure and layout.

To prevent these two goals from conflicting, OVG utilizes Orthogonal Projection. By projecting the high-frequency update into the "null-space" of the structural gradient, the method ensures that detail is only added in ways that do not alter the underlying shape or layout of the image.

Experimental Evaluation

We evaluated OVG on two challenging sparse-to-dense translation tasks: microscopy super-resolution (BBBC021) and sketch-to-image synthesis (Edges2Shoes).

Microscopy Super-Resolution

In the BBBC021 dataset, the model must perform extreme super-resolution (up to x32), hallucinating complex biological textures like cell membranes and nuclei from blurry input blobs. Standard DDIM produces only blurry approximations. In contrast, OVG restores the "Gaussianity" of the latent space, enabling the generation of sharp, scientifically plausible structures.

Figure 2: Qualitative comparison of super-resolution methods. OVG and EDM+OVG significantly outperform standard inversion and other baselines in restoring high-frequency biological detail.

Sketch-to-Image Synthesis

On the Edges2Shoes benchmark, OVG + EDM achieved state-of-the-art results, successfully synthesizing complex textures like leather grain and fabric from simple edge maps while maintaining significantly better structural alignment than stochastic baselines.

Discussion and Key Insights

Our analysis reveals several critical points regarding the nature of inversion:

  • Inversion Dynamics: Spectral collapse is a property of the inversion mathematics, not the specific neural architecture. It persists across both U-Net and Diffusion Transformer (DiT) models.
  • $x_0$-Prediction Advantage: Models trained to predict the clean image ($x_0$) rather than noise ($\epsilon$) are naturally more resistant to spectral collapse, though they are more prone to structural hallucination.
  • Pareto Frontier: OVG expands the Pareto frontier of the structure-texture trade-off, allowing for simultaneous maximization of realism and fidelity.

While OVG introduces some computational overhead due to gradient calculations during inversion, it provides a robust solution for high-fidelity unpaired image translation in spectrally sparse domains.

Summary updated

Abstract

Conditional diffusion inversion provides a powerful framework for unpaired image-to-image translation. However, we demonstrate through an extensive analysis that standard deterministic inversion (e.g. DDIM) fails when the source domain is spectrally sparse compared to the target domain (e.g., super-resolution, sketch-to-image). In these contexts, the recovered latent from the input does not follow the expected isotropic Gaussian distribution. Instead it exhibits a signal with lower frequencies, locking target sampling to oversmoothed and texture-poor generations. We term this phenomenon spectral collapse. We observe that stochastic alternatives attempting to restore the noise variance tend to break the semantic link to the input, leading to structural drift. To resolve this structure-texture trade-off, we propose Orthogonal Variance Guidance (OVG), an inference-time method that corrects the ODE dynamics to enforce the theoretical Gaussian noise magnitude within the null-space of the structural gradient. Extensive experiments on microscopy super-resolution (BBBC021) and sketch-to-image (Edges2Shoes) demonstrate that OVG effectively restores photorealistic textures while preserving structural fidelity.
BibTeX
@misc{bourriez2026spectral,
  author = {Nicolas Bourriez and Alexandre Vérine and Auguste Genovesio},
  title = {{Spectral Collapse in Diffusion Inversion}},
  year = {2026},
  eprint = {2602.13303},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.13303},
}
Figure