Characteristic Learning for Provable One Step Generation

Zhao Ding, Chenguang Duan, Yuling Jiao, Ruoxuan Li, Jerry Zhijian Yang, Pingwen ZhangarXiv 2024 · DOI · arXiv

Generative AI currently faces a fundamental trade-off between the rapid, one-step sampling of Generative Adversarial Networks (GANs) and the high-quality, stable training of diffusion and flow-based models. While diffusion models excel in quality, their iterative nature—often requiring hundreds of steps—creates a significant computational bottleneck.

A new research paper, "Characteristic Learning for Provable One Step Generation," introduces the "characteristic generator." This model distills the complex trajectories of flow-based models into a single, provably accurate neural network evaluation, effectively bridging the gap between speed and stability.

The Sampling Bottleneck

The primary hurdle for deploying state-of-the-art generative models on edge devices is the "sampling bottleneck." Models like Stable Diffusion transform noise into images by solving an Ordinary Differential Equation (ODE) step-by-step. While GANs offer a direct mapping, they are notoriously unstable and prone to mode collapse. Recent distillation methods attempt to compress diffusion models into one-step generators, but many lack a rigorous mathematical foundation. This work provides the first non-asymptotic convergence analysis for such models.

The Characteristic Learning Framework

The authors propose a three-phase approach termed Characteristic Learning:

  1. Velocity Matching: The model first estimates the "velocity field," which defines the directions particles must move to transform a simple Gaussian distribution into the target data distribution.
  2. Trajectory Generation: Using the estimated velocity, numerical solvers calculate discrete paths, or characteristics, that particles follow.
  3. Characteristic Fitting: A "student" neural network is trained to learn these paths. Instead of simulating the path incrementally, the student network learns to map the start of a trajectory directly to its end in a single step.

To maintain stability over long trajectories, the researchers introduce a semi-group penalty. This constraint ensures the generator is self-consistent: a single jump from the start to the end must match the composition of multiple smaller jumps along the same path.

Theoretical Breakthroughs: Bypassing the Curse of Dimensionality

The paper's most significant contribution is its theoretical grounding. It provides the first non-asymptotic convergence rate in the 2-Wasserstein distance for flow-based one-step models.

The Manifold Advantage

A key insight is that the model's convergence rate depends on the intrinsic dimension ($d^*$) of the data rather than the ambient pixel dimension ($d$). Since high-dimensional images typically lie on much lower-dimensional manifolds, this explains why the model remains efficient even at high resolutions.

Convergence rates under standard assumptions (ambient dimension d).
Convergence rates under the manifold hypothesis (intrinsic dimension d*).

As shown in the tables above, the transition from $d$ to $d^*$ in the exponent of the convergence rate represents a major theoretical leap, proving that these models can effectively "ignore" the high dimensionality of the pixel space by focusing on the underlying data structure.

Experimental Results

The researchers validated the Characteristic Generator (CG) across several benchmarks:

  • Efficiency and Quality: On CIFAR-10, the one-step CG achieved an FID of 4.59, comparable to traditional diffusion models (like DDPM) that require 1,000 steps.
  • Iterative Refinement: While optimized for one step, the model supports multi-step sampling. With just 4 steps, the FID on CIFAR-10 improves to 2.83, outperforming many GAN-based alternatives.
  • Scalability: The model successfully generated 512x512 images from the CelebA-HQ dataset using a self-distillation variant, demonstrating its applicability to high-resolution tasks.

Conclusion and Future Directions

By providing a rigorous mathematical bridge between flow-based stability and GAN-like speed, this research establishes a foundation for efficient, high-quality generative AI. While limitations remain—such as the "average-in-time" nature of the current bounds and singularities near the final data distribution—the characteristic generator moves the field closer to provably reliable one-step generation.

Summary updated

Abstract

We propose the characteristic generator, an one-step generative model that combines the sampling efficiency of generative adversarial networks (GANs) with the training stability of flow-based models. The proposed model is based on characteristics along which probability-density transport is governed by ordinary differential equations (ODEs). Specifically, we first estimate the underlying velocity field and numerically solve the probability-flow ODE using an exponential integrator, thereby obtaining discrete approximations of the characteristic trajectories. We then train a deep neural network to approximate these trajectories, yielding a one-step transport map from a Gaussian prior to the target distribution. Theoretically, we analyze the errors introduced by velocity-field estimation, numerical discretization, and characteristic approximation, and establish a nonasymptotic convergence rate in the Wasserstein-2 distance under mild assumptions on the data distribution. Moreover, under a low-dimensional linear-subspace assumption, we show that the convergence rate depends on the intrinsic data dimension rather than the potentially much larger ambient dimension, demonstrating the model's ability to alleviate the curse of dimensionality. Experiments on synthetic and real-world datasets show that the characteristic generator produces high-quality, high-resolution samples using only one or a few neural-network evaluations.
BibTeX
@misc{ding2024characteristic,
  author = {Zhao Ding and Chenguang Duan and Yuling Jiao and Ruoxuan Li and Jerry Zhijian Yang and Pingwen Zhang},
  title = {{Characteristic Learning for Provable One Step Generation}},
  year = {2024},
  eprint = {2405.05512},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.05512},
}
Figure