Uncovering the Generative Landscape of Diffusion Models: A Spectral-Guided Quality-Diversity Diffusion Framework

Hannah Janmohamed, Joao Correia, Penousal Machado, Antoine CullyarXiv 2026 · arXiv

This paper introduces Spectral-Guided Quality-Diversity Diffusion, a framework that combines evolutionary search with diffusion models to uncover the full expressive range of generative AI. By leveraging a novel mutation operator based on spectral analysis of the model's internal activations, the authors demonstrate a way to generate portfolios of images that are both high-quality and semantically diverse, significantly outperforming standard sampling techniques.

TL;DR

Standard diffusion models often get stuck in "safe" high-likelihood modes, producing repetitive results. This paper introduces a Quality-Diversity (QD) framework and a novel spectral mutation operator that navigates the model's latent space along semantically sensitive directions to discover a diverse array of high-performing outputs.

What Problem Does This Paper Address?

Denoising Diffusion Probabilistic Models (DDPMs) are the current state-of-the-art for image synthesis, but they have a hidden bias: they tend to sample from high-likelihood regions of the learned distribution. In practice, this means that if you give a model a prompt like "a street with several cars," different random seeds often produce images with nearly identical global structures.

In many fields—from creative arts to drug discovery—a single "most likely" answer isn't enough. We need a "diverse portfolio" of high-quality solutions. Traditional Quality-Diversity (QD) algorithms, which are designed to find diverse sets of high-performing solutions, struggle with the high-dimensional latent and feature spaces typical of modern generative models. This paper bridges that gap.

Methods

The authors propose a two-part solution:

  1. QD-Diffusion Framework: This adapts the Dominated Novelty Search (DNS) algorithm to work with diffusion models. Instead of hand-crafting low-dimensional features for diversity, they use high-dimensional embeddings from foundation models (like DreamSim) as a proxy for semantic difference. Crucially, they add a fitness thresholding mechanism: a candidate is only archived if its quality (measured by CLIP score) exceeds a certain bar, preventing the search from drifting into "hallucinated" or corrupted regions of the latent manifold.
  2. h-space Spectral Mutation Operator: This is the core technical innovation. Traditional evolutionary mutations (like adding Gaussian noise) are inefficient in high-dimensional spaces. The authors analyze the Jacobian of the mapping from the latent state to the model's internal bottleneck activations (the "h-space").
  3. By computing the principal singular vectors of this Jacobian, they identify the directions of maximal semantic sensitivity.
  4. Perturbing the latent state along these axes produces meaningful semantic shifts rather than just random pixel noise.
  5. They use deterministic inversion (via DDIM or Euler schedulers) to map these intermediate-timestep mutations back to the initial noise space, keeping the "genotype" consistent for the evolutionary archive.
The Quality-Diversity Diffusion Framework loop.

Datasets

The framework was evaluated on text-to-image generation using:

  • Parti Prompts: A benchmark of 36 diverse prompts (for SDXL-Turbo) and a subset of 12 (for Flux.1) covering categories like "Abstract," "World Knowledge," and "Animals."
  • MS-COCO 2017: Used as the reference distribution for calculating the Fréchet Inception Distance (FID).

Key Insights and Ideas

  • The Geometry of the Bottleneck: The paper confirms that the "h-space" (bottleneck layer) of diffusion models preserves a local linear structure that corresponds to semantic features. By targeting this layer for spectral analysis, the mutation operator becomes "architecture-agnostic," working on both U-Net and Transformer-based (DiT) backbones.
  • Intermediate Timesteps Matter: The authors found that semantic editing is most effective at intermediate timesteps ($t \approx 0.5T$ to $0.7T$). Mutating at the very beginning of the diffusion process ($t=T$) rarely yields meaningful semantic shifts.
  • Novelty vs. Validity: In generative tasks, pure novelty search is dangerous. Without the fitness thresholding introduced here, QD algorithms tend to populate archives with "novel" but visually corrupted images that the model doesn't actually "know" how to render correctly.
Visualizing the h-space Spectral Mutation Operator.

Key Results

The authors compared their method against 12 baselines, including standard random sampling, gradient-based optimization (Adam), and other QD variants like CVT-ME and CMA-MAE.

  • Superior Quality and Diversity: The Spectral-Guided approach achieved the lowest FID scores on both SDXL-Turbo (81.90) and Flux.1 (137.58).
  • Pareto Dominance: In head-to-head comparisons, the method consistently dominated baselines in the trade-off between total fitness and semantic diversity.
  • Qualitative Impact: As seen in Figure 1, while random seeds converge to structurally similar modes, the proposed method uncovers distinct interpretations—varying camera angles, compositions, and lighting—while maintaining strict prompt alignment.
Comparison of Quality vs. Diversity Pareto fronts for SDXL-Turbo.

Limitations and Future Work

  • Computational Cost: Calculating the Jacobian and its singular vectors is significantly more expensive than simple noise addition. On the Flux.1 model, the per-image generation time was roughly 2.5x slower than some baselines.
  • Approximation: The authors suggest that finding cheaper approximations for the spectral decomposition is a key direction for making this real-time.
  • Beyond Images: While tested on images, the framework is domain-agnostic. The authors highlight its potential for scientific discovery, such as designing diverse sets of proteins or materials where "mode-seeking" behavior in models might miss viable, non-obvious solutions.
  • Safety: The authors note that uncovering "hidden" diversity could potentially bypass safety filters designed to suppress specific types of content, necessitating robust post-generation filtering.

Summary updated

Abstract

Denoising Diffusion Probabilistic Models have emerged as powerful generative priors across domains ranging from molecular design to high-resolution image synthesis. However, standard diffusion sampling is inherently biased toward high-likelihood regions, often limiting the diversity of outputs. In many domains, a single, high-likelihood sample is insufficient. Instead, diverse portfolios of high-performing solutions are required, motivating methods that uncover a model's full expressive range. Here we show that integrating Quality-Diversity (QD) optimisation with diffusion models enables the discovery of diverse, high-performing outputs. We first propose Quality-Diversity Diffusion, a framework that adapts QD to diffusion models by directly optimising the model's latent noise and conditioning embeddings and leveraging embedding models to quantify diversity. Second, we introduce a novel h-space Spectral Mutation operator, which guides search along directions of maximal semantic sensitivity by analysing principal singular vectors of the model's input-to-activation Jacobian. We evaluate our combined approach, Spectral-Guided Quality-Diversity Diffusion, on text-to-image generation. Against 12 baselines, our method consistently achieves the lowest FID score, demonstrating an excellent balance of image quality and diversity.
BibTeX
@misc{janmohamed2026uncovering,
  author = {Hannah Janmohamed and Joao Correia and Penousal Machado and Antoine Cully},
  title = {{Uncovering the Generative Landscape of Diffusion Models: A Spectral-Guided Quality-Diversity Diffusion Framework}},
  year = {2026},
  eprint = {2610.15724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.15724},
}
Figure