Complexity-Balanced Diffusion Splitting

Noam Issachar, Dani Lischinski, Raanan FattalarXiv 2026 · arXiv

TL;DR

Complexity-Balanced Splitting (CBS) optimizes diffusion models by dividing the generation timeline into intervals of equal modeling difficulty and assigning specialized sub-networks to each. This principled approach significantly boosts image synthesis quality (up to a 35% FID improvement) without increasing per-step inference FLOPs or active parameter counts.

What Problem Does This Paper Address?

Standard diffusion and flow-matching models rely on a single, "monolithic" neural network to handle the entire generation process. However, the generative dynamics change drastically over time: the model starts by forming coarse structures from pure noise and ends by refining high-frequency details near the data manifold.

Deploying the same massive network uniformly across all timesteps is highly inefficient. While previous works tried training specialized "expert" networks for different time intervals, they relied on arbitrary uniform splits or computationally expensive search procedures. This paper addresses the lack of a principled, efficient criterion for distributing network capacity across the generation timeline.

Methods

The authors introduce Complexity-Balanced Splitting (CBS), a search-free framework grounded in approximation theory and de Boor’s equidistribution principle. Instead of splitting time uniformly, CBS partitions the timeline so that each sub-network faces an equal "approximation burden."

To measure local complexity over time, they propose two tractable monitor functions:

  1. Dirichlet Spectral Energy: A spatial roughness metric that bounds the network's approximation error based on the flow field's gradients, computed efficiently using randomized trace estimators.
  2. Path Acceleration: A geometric metric measuring the second-order time derivative (curvature) of the sampling trajectories.

Crucially, these complexity profiles can be computed upfront using a lightweight, under-trained auxiliary model (e.g., trained on just 10% of the data), introducing negligible computational overhead.

Datasets

The method was evaluated across three distinct generative environments:

  • ImageNet-256: Evaluated in latent space using the Scalable Interpolant Transformer (SiT).
  • ImageNet-64: Evaluated in pixel space using the Just Image Transformer (JiT) to test raw, uncompressed spatial gradients.
  • CIFAR-10: Evaluated unconditionally using a standard convolutional UNet backbone.

Key Insights and Ideas

  • Errors Accumulate Forward in Time: In path-integrated generation, an early mistake causes the trajectory to permanently diverge. Minimizing the maximum local error rather than the average error across time is key to better sampling quality. Maximal local errors govern ODE approximation
  • Complexity is Non-Linear and Robust: Trajectory complexity is highly non-linear and unique to each dataset. Remarkably, this curve can be accurately captured by a tiny, cheap auxiliary model, meaning you don't need to train a full-scale model just to find the optimal split points.
  • Geometry Beats Pure Flow: Path acceleration slightly outperforms Dirichlet energy because it directly models the actual sampling trajectory rather than just the static velocity field.
Figure 1: Maximal local errors govern ODE approximation.

Key Results

  • Zero Extra Inference Cost: Because only one sub-network is active at any given timestep, CBS scales total parameter capacity without increasing per-step FLOPs.
  • Significant Quality Gains: On ImageNet-256 with SiT-XL and Classifier-Free Guidance (CFG), CBS improves the Fréchet Inception Distance (FID) by ~35% (dropping from 6.24 to 4.03) compared to naive uniform splitting. Quantitative evaluation of SiT on ImageNet-256
  • Architecture Agnostic: CBS consistently outperforms monolithic baselines and uniform splits across Transformers (SiT, JiT) and CNNs (UNet).
Table 1: Quantitative evaluation of SiT on ImageNet-256. We compare the standard monolithic baseline, a naive uniformly partitioned ensemble (partitioned at 0.33, 0.66), and our approach (CBS) using complexity-derived boundaries (0.4, 0.77). Across all model capacities (S/2, B/2, and XL/2), our method yields consistent and significant improvements in FID, Inception Score (IS), Recall and Precision, both with and without Classifier-Free Guidance (CFG). Crucially, CBS achieves these generative gains while strictly maintaining the exact same active parameter count and per-step inference cost (GFLOPs) as the standard monolithic baseline.

Limitations and Future Work

  • Temporal Only: The current formulation only splits capacity across the time dimension.
  • Future Directions: The authors suggest extending the equidistribution principle to other domains, such as spatial splitting, where individual image tokens could be dynamically routed to different specialized networks based on local spatial complexity.

Summary updated

Abstract

Standard continuous-time generative models rely on monolithic architectures that must navigate vastly different signal regimes, from isotropic noise to intricate data distributions. While scaling model capacity improves performance, deploying a massive network uniformly across the entire generative timeline is inherently inefficient. In this work, we propose Complexity-Balanced Splitting (CBS), a principled framework for temporal capacity allocation that distributes the generative workload across multiple specialized sub-networks. Grounded in function approximation theory and de Boor's equidistribution principle, CBS partitions the diffusion timeline into segments of equal approximation burden, allocating more representational capacity to regions where the generative dynamics are more difficult to model. To estimate this local complexity, we introduce two complementary and tractable monitor functions: a spatial measure based on the flow's Dirichlet energy, and a geometric measure based on the acceleration of the sampling trajectories. Using a lightweight auxiliary model to estimate these complexity profiles, our approach eliminates the need for heuristic temporal splits or computationally expensive search procedures. Extensive evaluation across multiple architectures (SiT, JiT, and UNet) and datasets demonstrates that CBS consistently improves synthesis quality without increasing per-step inference cost. In particular, CBS improves FID by ~35% on SiT-XL with CFG relative to naive temporal partitioning. Project page is available at https://noamissachar.github.io/CBS/.
BibTeX
@misc{issachar2026complexity,
  author = {Noam Issachar and Dani Lischinski and Raanan Fattal},
  title = {{Complexity-Balanced Diffusion Splitting}},
  year = {2026},
  eprint = {2606.06477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.06477},
}
Figure