TL;DR
Complexity-Balanced Splitting (CBS) optimizes diffusion models by dividing the generation timeline into intervals of equal modeling difficulty and assigning specialized sub-networks to each. This principled approach significantly boosts image synthesis quality (up to a 35% FID improvement) without increasing per-step inference FLOPs or active parameter counts.
What Problem Does This Paper Address?
Standard diffusion and flow-matching models rely on a single, "monolithic" neural network to handle the entire generation process. However, the generative dynamics change drastically over time: the model starts by forming coarse structures from pure noise and ends by refining high-frequency details near the data manifold.
Deploying the same massive network uniformly across all timesteps is highly inefficient. While previous works tried training specialized "expert" networks for different time intervals, they relied on arbitrary uniform splits or computationally expensive search procedures. This paper addresses the lack of a principled, efficient criterion for distributing network capacity across the generation timeline.
Methods
The authors introduce Complexity-Balanced Splitting (CBS), a search-free framework grounded in approximation theory and de Boor’s equidistribution principle. Instead of splitting time uniformly, CBS partitions the timeline so that each sub-network faces an equal "approximation burden."
To measure local complexity over time, they propose two tractable monitor functions:
- Dirichlet Spectral Energy: A spatial roughness metric that bounds the network's approximation error based on the flow field's gradients, computed efficiently using randomized trace estimators.
- Path Acceleration: A geometric metric measuring the second-order time derivative (curvature) of the sampling trajectories.
Crucially, these complexity profiles can be computed upfront using a lightweight, under-trained auxiliary model (e.g., trained on just 10% of the data), introducing negligible computational overhead.
Datasets
The method was evaluated across three distinct generative environments:
- ImageNet-256: Evaluated in latent space using the Scalable Interpolant Transformer (SiT).
- ImageNet-64: Evaluated in pixel space using the Just Image Transformer (JiT) to test raw, uncompressed spatial gradients.
- CIFAR-10: Evaluated unconditionally using a standard convolutional UNet backbone.
Key Insights and Ideas
- Errors Accumulate Forward in Time: In path-integrated generation, an early mistake causes the trajectory to permanently diverge. Minimizing the maximum local error rather than the average error across time is key to better sampling quality. Maximal local errors govern ODE approximation
- Complexity is Non-Linear and Robust: Trajectory complexity is highly non-linear and unique to each dataset. Remarkably, this curve can be accurately captured by a tiny, cheap auxiliary model, meaning you don't need to train a full-scale model just to find the optimal split points.
- Geometry Beats Pure Flow: Path acceleration slightly outperforms Dirichlet energy because it directly models the actual sampling trajectory rather than just the static velocity field.
Key Results
- Zero Extra Inference Cost: Because only one sub-network is active at any given timestep, CBS scales total parameter capacity without increasing per-step FLOPs.
- Significant Quality Gains: On ImageNet-256 with SiT-XL and Classifier-Free Guidance (CFG), CBS improves the Fréchet Inception Distance (FID) by ~35% (dropping from 6.24 to 4.03) compared to naive uniform splitting. Quantitative evaluation of SiT on ImageNet-256
- Architecture Agnostic: CBS consistently outperforms monolithic baselines and uniform splits across Transformers (SiT, JiT) and CNNs (UNet).
Limitations and Future Work
- Temporal Only: The current formulation only splits capacity across the time dimension.
- Future Directions: The authors suggest extending the equidistribution principle to other domains, such as spatial splitting, where individual image tokens could be dynamically routed to different specialized networks based on local spatial complexity.
Summary updated