Role-Separated Diffusion Transformers: Global Rule Isolation and Deterministic Sampling Manifold Compression

Zhiguang Liu, Yi ShangarXiv 2026 · arXiv

TL;DR

RoSeDiT (Role-Separated Diffusion Transformer) improves image generation by decoupling "global rules" (like class and noise level) from "image content" (the actual pixels/tokens). By using dedicated controller tokens and a novel "listening" mechanism for layer normalization, it achieves state-of-the-art performance with significantly fewer parameters and faster training than standard Diffusion Transformers.

What Problem Does This Paper Address?

In the world of Diffusion Transformers (DiT), models typically use Adaptive Layer Normalization (adaLN) to inject global information—such as what class of image to generate and the current noise level—into every image patch.

The authors argue this creates a representational bottleneck. In standard architectures, these global "rules" are static and spread across all image tokens, leading to high representational complexity (entropy). Drawing inspiration from visual reasoning research, the authors suggest that a model should ideally separate the "controller" (the part that understands the global rule) from the "content" (the part that represents spatial image data). Without this separation, the model wastes capacity re-learning the same global rules at every spatial location.

Methods

The authors introduce RoSeDiT, which implements "role separation" through three key components:

  1. Condition-Initialized Controller Tokens: Instead of just using a single vector for conditioning, RoSeDiT prepends 12 dedicated "controller tokens" (8 for class, 4 for timestep) to the image tokens. These tokens evolve as they pass through the transformer blocks, accumulating information from the image data.
  2. The adaLN Listening Mechanism: This is the core innovation. In standard DiTs, adaLN only looks at fixed embeddings. In RoSeDiT, the adaLN layers "listen" to the evolving state of the controller tokens. This allows the global conditioning to dynamically adapt to the intermediate features of the image being generated.
  3. Parameter-Shared Structural Pass: To reinforce the role of the controllers, the authors add a sparse attention pass where image tokens only look at their immediate 3x3 neighborhood and the controller tokens. To keep the model efficient, this pass reuses the weights from the standard dense attention layer, adding virtually zero extra parameters.

Additionally, they introduce Deterministic Sampling Manifold Compression (DSMC). This is a two-stage training schedule that reuses specific (image, noise level, noise) triples. They discovered that models with identical denoising losses can have wildly different image quality (FID scores), and DSMC helps "compress" the model's outputs onto the actual image manifold more effectively.

Datasets

The model was trained and evaluated on the standard ImageNet 256×256 benchmark. The authors compared their results against established baselines like DiT, SiT (Scalable Interpolant Transformers), and REPA (Representation Alignment).

Key Insights and Ideas

  • Entropy Reduction: By centralizing the "rule" in controller tokens, the theoretical representational complexity drops from $O(Ld)$ (where $L$ is the number of tokens) to $O(d)$. This makes the model much more efficient at learning global transformations.
  • The "Listening" Requirement: Simply adding extra tokens isn't enough. The authors found that the normalization layers must be able to read from these tokens (the "listening" mechanism) to break the bottleneck and see significant performance gains.
  • Loss $\neq$ Quality: A major "aha moment" was observing that two models with the same denoising loss could have a large gap in FID. This implies that standard training doesn't perfectly align the model with the "image manifold." DSMC was designed specifically to bridge this gap by mixing fresh stochastic samples with deterministic re-runs.

Key Results

RoSeDiT consistently outperforms standard Diffusion Transformers across all scales:

  • Efficiency: RoSeDiT-L/2 (458M parameters) outperformed the much larger SiT-XL/2 (675M parameters), achieving a better FID score with 32% fewer parameters.
  • Speed: RoSeDiT-XL/2 reached an FID of 7.98 in just 1.4M training steps. For comparison, the baseline SiT-XL/2 took 7M steps to reach a worse score (8.3).
  • State-of-the-Art: When combined with the DSMC training schedule, RoSeDiT-XL/2 achieved an FID of 5.84, surpassing the performance of SiT + REPA (5.9) while using significantly fewer training epochs.

Limitations and Future Work

  • Evaluation Consistency: The authors note that ImageNet self-evaluation can be inconsistent between training and validation sets, making very low FID scores (below 2.23) potentially misleading.
  • Architecture Specificity: While the "listening" mechanism is powerful, it is currently tailored for the adaLN structure common in DiTs. Extending this to other transformer variants (like those used in LLMs) remains an open question.
  • Scaling: While the results at the XL scale are impressive, the authors suggest that even larger models or different training objectives (like combining with REPA at scale) could yield further improvements.

Summary updated

Abstract

We present Role-Separated Diffusion Transformers (RoSeDiT), a new diffusion transformer architecture that isolates the global rule from the image tokens. RoSeViT showed in visual reasoning that a vanilla ViT spreads a shared rule over all $L$ tokens at an entropy of order $Ld$, whereas an enforced controller route reduces it to order $d$. In class-conditional DiT and SiT, the class and the noise level are explicitly conditioned by adaLN, so a RoSeViT-style controller channel on the ViT side is insufficient to establish the full role separation. By introducing the adaLN listening mechanism, which makes adaLN compute under the controller's command, RoSeDiT brings the entropy theorem and role separation into the latent diffusion domain. Under the same SiT training and evaluation protocol at the same level of parameter count, RoSeDiT surpasses SiT at all four model scales at 400K steps, and RoSeDiT-L/2 even surpasses SiT-XL/2's 17.2 FID with 15.34 and 32% fewer parameters. RoSeDiT-XL/2 surpasses SiT-XL/2's 7M-step FID of 8.3 at 1.4M steps with 7.98, using one-fifth as many training steps. We further observe that models with the same denoising loss can have a large FID gap, which indicates a misalignment between the denoising objective and the image manifold. We formalize this phenomenon into Deterministic Sampling Manifold Compression (DSMC), a two-stage training schedule with which RoSeDiT-XL/2 surpasses SiT + REPA's 4M-step FID of 5.9 at 2.4M steps with 5.84.
BibTeX
@misc{liu2026role,
  author = {Zhiguang Liu and Yi Shang},
  title = {{Role-Separated Diffusion Transformers: Global Rule Isolation and Deterministic Sampling Manifold Compression}},
  year = {2026},
  eprint = {2611.19009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19009},
}
Figure