Deep generative models, particularly diffusion and autoregressive frameworks, have revolutionized our ability to synthesize complex data. However, traditional architectures suffer from a fundamental operational bottleneck: sampling step rigidity.
When a model is trained, its generation process is typically coupled to a fixed, discrete sequence of steps or a hard-coded temporal grid. At inference time, this lock-in prevents the model from dynamically adjusting its computational budget. If an application requires a fast, low-compute preview, the model cannot easily drop steps without catastrophic quality degradation. Conversely, if extra computational resources are available, the model cannot easily leverage them to perform finer, multi-step refinement.
To solve this, a major research direction focuses on decoupling training schedules from inference schedules, enabling true inference-time sampling flexibility.
The Grid Lock: Why Discrete Discretization Fails
To understand why traditional models are rigid, consider how standard diffusion models are trained. They learn to reverse a degradation process by predicting the state at step $t-1$ given the state at step $t$. Because the transition distributions are parameterized explicitly for a fixed set of integer timesteps, the model becomes trapped on this pre-defined grid.
As Daniel Watson et al. demonstrate, this grid lock severely limits inference optimization (Learning to Efficiently Sample from Diffusion Probabilistic Models). When search algorithms attempt to find more efficient inference paths, they are restricted to the original training grid, preventing the discovery of superior intermediate transition points that exist between the integer steps. Similarly, Bocheng Li et al. show that post-hoc optimization techniques like temporal pruning are fundamentally bounded by the maximum number of steps established during initial training (Few-shot Temporal Pruning Accelerates Diffusion Models for Text Generation).
This rigidity becomes even more pronounced when models are compressed or applied to structured domains:
- Quantization Collapse: Junhyuk So et al. observe that in quantized diffusion models, internal activation distributions shift drastically across steps. If the inference schedule deviates from the training configuration, the quantized parameters fail to adapt, causing severe performance collapse (Temporal Dynamic Quantization for Diffusion Models).
- Structural Constraints: In molecular conformation generation, Jiying Zhang et al. highlight that standard frameworks force the number of inference iterations to match training steps, severely limiting the speed and flexibility needed to scale up drug discovery pipelines (SubgDiff: A Subgraph Diffusion Model to Improve Molecular Representation Learning).
Breaking the Grid via Continuous-Time Frameworks
The most mathematically rigorous strategy to eliminate sampling rigidity is to shift from discrete-time steps to a continuous-time framework. This transition is particularly powerful for discrete data spaces (such as text, graphs, or categorical features), where models traditionally relied on fixed jump schedules.
Instead of learning step-by-step transition probabilities, continuous-time models parameterize the rate of change using Continuous Time Markov Chains (CTMCs) or stochastic jump processes. Once the model learns these continuous transition rates, users can integrate the process at inference time using any numerical solver and any arbitrary step size.
Discrete-Time Grid (Rigid):
[Step 1] ----> [Step 2] ----> [Step 3] ----> [Step 4] (Cannot sample in between)
Continuous-Time Flow (Flexible):
[t = 0] ------------------------------------> [t = 1] (Can sample at any t_i)
Andrew Campbell et al. established this foundation for discrete data in A Continuous Time Framework for Discrete Denoising Models, unlocking adaptive sampling techniques that were previously restricted to continuous domains. To make these continuous formulations robust when accelerated into very few steps, Haoran Sun et al. introduced a categorical ratio matching objective that preserves generation quality even under highly compressed inference budgets (Score-based Continuous-time Discrete Diffusion Models).
This paradigm shift has transformed structured data generation:
- Graph Generation: Frameworks like DISCO, developed by Zhe Xu et al. (Discrete-state Continuous-time Diffusion for Graph Generation), and Cometh, developed by Antoine Siraudin et al. (Cometh: A continuous-time discrete-state graph diffusion model), decouple training from sampling schedules entirely, allowing users to freely trade off generation speed for structural accuracy post-training.
- Protein Co-Design: Andrew Campbell et al. extended continuous-time modeling to multi-modal spaces with Discrete Flow Models, allowing joint generation of protein sequences and structures with unprecedented runtime control over the balance between sample quality and diversity (Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design).
Flexible Distillation and Trajectory Mapping
Another approach to bypassing step rigidity involves restructuring how models map generation trajectories. Traditional distillation methods accelerate diffusion models by compressing them into a fixed small number of steps (e.g., exactly two or four iterations). However, as Zhenyu Zhou et al. point out, this merely replaces one rigid constraint with another, preventing the model from dynamically adjusting its compute budget at runtime (Simple and Fast Distillation of Diffusion Models). A similar limitation applies to strict single-step architectures like SwiftBrush by Thuan Hoang Nguyen et al., which lack the flexibility to perform multi-step refinement when extra compute is available (SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation).
To achieve true flexibility, Consistency Models, introduced by Mark Chen et al., enforce a property where any point along a smooth "Probability Flow" trajectory maps directly back to the data origin (Consistency Models).
Because the model learns the entire continuous trajectory rather than isolated steps, it can support ultra-fast single-step generation while retaining the ability to perform multi-step iterative refinement.
Consistency Trajectory:
Any point x_t on the path can be mapped directly to x_0.
If more compute is available: x_t -> x_0 -> add noise -> x_t' -> x_0 (Refinement)
For downstream tasks like image super-resolution, Zongsheng Yue leverages this trajectory flexibility via a Partial Noise Prediction strategy (Arbitrary-steps Image Super-resolution via Diffusion Inversion). Instead of adhering to a hard-coded budget, the system dynamically scales its sampling effort between one and five steps based on the specific degradation level of the input image.
Beyond Diffusion: Flexibility in Autoregressive Models
The challenge of sampling step rigidity is not unique to diffusion models; it also restricts traditional visual autoregressive (VAR) frameworks. Standard VAR models generate images scale-by-scale using rigid, hard-coded schedules optimized for specific resolutions.
To break this constraint, Siyu Jiao et al. introduced FlexVAR, which abandons traditional residual prediction in favor of predicting ground-truth images at every scale (FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction). This architectural shift allows a single model to adapt dynamically to varying inference schedules, aspect ratios, and anytime preview requirements.
Similarly, Hermann Kumbong et al. developed a hierarchical masked auto-regressive framework (HMAR) that treats generation as a Markovian process across scales (HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation). By decoupling the generation steps from a fixed resolution grid, HMAR allows the model to dynamically trade off generation speed for higher perceptual quality at runtime, bringing the same inference-time flexibility enjoyed by continuous-time diffusion models to the autoregressive domain.
Key Foundational Work
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021) — contributed 2 concept elements to this direction.
A Continuous Time Framework for Discrete Denoising Models (2022) — contributed 1 concept elements to this direction.