Introduction
Consistency Models (CMs) represent a significant leap in generative modeling, offering the ability to generate high-quality samples in just one or two inference steps. However, training these models from scratch—a process known as Consistency Training (CT)—remains a complex challenge. This paper identifies a critical bottleneck in current CT methodologies: the reliance on log-normal noise scheduling, which tends to under-expose models to high-level noise regimes.
The High-Noise Deficit
The standard approach in Consistency Training utilizes a log-normal noise scheduler. While effective for refining fine-grained details, this distribution tends to "over-sample" low-level noise. Consequently, the model remains under-exposed to high-level noise—the high-variance state where global image structure is first established. Without sufficient training at these levels, the model struggles to bridge the gap from pure noise to coherent structure, leading to instability and degraded image quality.
Proposed Methodology
To address these limitations, the authors introduce three key refinements to the Consistency Training pipeline: polynomial noise distribution, predefined noise vectors, and a sinusoidal curriculum.
Polynomial Noise Distribution
Instead of the standard log-normal curve, the researchers propose a polynomial noise distribution. This shift ensures a more balanced exposure across the noise spectrum, providing the model with a healthy mix of structural (high-level) and detail-oriented (low-level) noise.
Experiments indicated that a polynomial function with a power of 4 offers the optimal balance for training stability and performance, significantly increasing the frequency of high-noise samples compared to the baseline.
Predefined Karras Noise Vector
To enhance stability, the authors decouple noise levels from the number of discretization steps. By pre-calculating a fixed vector of noise levels based on the Karras algorithm, the model is provided with consistent "anchors" throughout the training process, regardless of changes in discretization. This prevents the generation of unpredictable noise levels as the curriculum evolves.
Sinusoidal Curriculum
Traditional curricula often use step-wise increases in discretization steps ($N$), which can introduce sudden shocks to the training process. The authors propose a sinusoidal curriculum that smoothly increases and then decreases $N$. This gradual transition allows the model to adapt to increasing complexity without the instability associated with abrupt step changes.
Results and Discussion
The effectiveness of these refinements was evaluated on the CIFAR-10 dataset. The proposed polynomial approach achieved a Fréchet Inception Distance (FID) score of 30.48. In comparison, models trained with standard log-normal distributions under similar low-depth architectures yielded significantly higher (worse) scores, often exceeding 40 or 60.
A key finding is the robustness of this method even in low-depth models. By prioritizing high-noise training, the authors demonstrated that simpler architectures could outperform deeper models trained with traditional scheduling. This suggests that the quality of the noise schedule is as important as the capacity of the model itself.
Conclusion
The central insight of this work is that high-level noise is a critical training phase, not merely a starting point. By refining noise distributions and curricula to prioritize these structural phases, Consistency Models can achieve greater stability and superior image quality. Future research will focus on scaling these principles to higher-resolution datasets and more complex architectures to determine if these performance gains hold at scale.
Summary updated