Optimal Linear Subspace Search: Learning to Construct Fast and High-Quality Schedulers for Diffusion Models

Zhongjie Duan, Chengyu Wang, Cen Chen, Jun Huang, Weining Qian

Introduction

Diffusion models have revolutionized image synthesis, achieving state-of-the-art results in visual fidelity. However, their primary drawback remains the high computational cost: generating a single image typically requires hundreds of iterative denoising steps. While existing schedulers like DDIM or DPM-Solver attempt to accelerate this process by skipping steps, they often suffer from significant quality degradation when pushed to extremely low step counts (e.g., fewer than 10 steps).

This paper introduces Optimal Linear Subspace Search (OLSS), a data-driven approach to scheduler design. By treating the generation process as an expansion within a linear subspace, OLSS enables high-quality image synthesis in as few as 5 steps—achieving sub-second inference on modern hardware without requiring expensive model retraining or distillation.

The Challenge of Iterative Denoising

Traditional acceleration methods generally fall into two categories: mathematical ODE/SDE solvers and model distillation. Mathematical solvers approximate the denoising path using general numerical theories, which can become inaccurate at high speeds. Distillation involves training a new, smaller model to mimic the original, a process that is both computationally intensive and difficult to scale.

OLSS seeks a middle ground: a lightweight, training-free scheduler that "learns" the optimal shortcut through the generation process by observing the specific behavior of a pre-trained model.

The core innovation of OLSS lies in its shift from fixed mathematical formulas to a data-driven coefficient learning strategy.

1. Linear Subspace Analysis

The authors demonstrate that the generation process can be accurately modeled as the expansion of a linear subspace spanned by the initial noise and previous model outputs. Analysis of intermediate variables reveals high correlation and redundancy, suggesting that the complex, non-linear path of a diffusion model can be approximated within a lower-dimensional linear space.

Geometric representation of the generation process and the orthogonal projection in the linear subspace.

2. Coefficient Learning and Path Optimization

Instead of relying on hard-coded calculus coefficients, OLSS uses a simple linear model to predict the next state. The process involves:

  • Calibration: Running a full 1000-step generation on a small sample (e.g., 32 images) to collect ground-truth data.
  • Least Squares Optimization: Solving for coefficients that minimize approximation error for a target step count.
  • Binary Search Pathfinding: Identifying the optimal sequence of time steps (the "path") to minimize cumulative error, particularly in early, high-risk stages of generation.

Experimental Evaluation

The researchers evaluated OLSS using Stable Diffusion (v1.4 and v2.1) across open-domain prompts and fine-tuned datasets like CelebA-HQ and LSUN-Church.

Key Results

  • High-Speed Quality: At 5 and 10 steps, OLSS consistently outperformed standard schedulers like DDIM and DPM-Solver++ in terms of Frechet Inception Distance (FID).
  • Efficiency: The method achieves high-quality results in under one second on an NVIDIA A100 GPU.
  • Visual Fidelity: Unlike traditional solvers that may produce blurry results at low step counts, OLSS preserves fine textures and details, making 5-step outputs nearly indistinguishable from 1000-step benchmarks.
FID scores and inference time as a function of the number of sampling steps.

Conclusion

Optimal Linear Subspace Search (OLSS) demonstrates that the "shortcut" to fast diffusion is linear. By leveraging a small amount of calibration data, OLSS finds more efficient denoising paths than general-purpose mathematical solvers. While it requires a one-time calibration step, its ability to provide high-fidelity images at extreme speeds without model retraining makes it a powerful tool for deploying diffusion models in real-time applications.

Summary updated

BibTeX
@inproceedings{duan2023optimal,
  author = {Zhongjie Duan and Chengyu Wang and Cen Chen and Jun Huang and Weining Qian},
  title = {{Optimal Linear Subspace Search: Learning to Construct Fast and High-Quality Schedulers for Diffusion Models}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2023},
  url = {https://openreview.net/forum?id=-FKDdIObeA},
}
Figure