Abstract
Score-based generative models exhibit inherent architectural biases that favor specific data directions. We introduce Score Anisotropy Directions (SADs) to quantify these biases at initialization. Our findings reveal that data aligned with directions corresponding to small eigenvalues of the network's average geometry are significantly easier to learn, providing a predictive tool for model performance before training begins.
Introduction
In deep learning, neural networks possess inductive biases that facilitate generalization. While these biases are well-understood in discriminative models through Neural Anisotropy Directions (NADs), a cohesive theory for generative modeling—specifically score-based diffusion—has been lacking. This paper addresses why architectures like U-Nets learn certain image features more effectively than others by quantifying the interaction between network geometry and data.
Quantifying Architectural Geometry
The authors propose a framework centered on the Average Geometry ($G_{\mathcal{F}}$) of a network, defined as the expected outer product of the network’s output at initialization. The eigenvectors of this matrix, termed Score Anisotropy Directions (SADs), represent an ordered set of directions in the output space that the model is "predisposed" to generate.
A core hypothesis of this work is that data aligned with SADs corresponding to small eigenvalues is significantly easier for the model to learn. Theoretical grounding (Theorem 1) suggests that in linear models, Stochastic Gradient Descent (SGD) is more stable in these directions due to lower gradient noise.
Key Insights: Harmonic Bias and Architecture
The research reveals that different architectures induce distinct geometric biases. U-Nets, the standard for convolutional diffusion, exhibit a "harmonic" bias where their SADs resemble Fourier or Wavelet bases. This explains their natural proficiency in capturing specific spatial frequencies.
In contrast, Diffusion Transformers (DiT) do not induce such harmonic representations. Their geometry is characterized by high eigen-multiplicity, reflecting a "looser" inductive bias. Furthermore, minor implementation details—such as resampling methods or padding—can create significant directional biases that impact learning.
Empirical Results
The theory was validated using synthetic rank-one manifolds and standard benchmarks like MNIST and CelebA-HQ. By rotating the MNIST dataset to align with the "best" SADs (minimizing eigenvalue alignment), the authors achieved a Wasserstein distance of 0.61. Conversely, aligning with the "worst" directions caused the error to skyrocket to 8.52, leading to severe artifacts and mode collapse.
Conclusion
SADs provide a fundamental lens through which to understand the learning dynamics of score-based models. This work opens avenues for more efficient Neural Architecture Search (NAS) and AutoML, potentially allowing researchers to "score" an architecture's suitability for a dataset without the need for expensive training cycles.
Summary updated