ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images

Adrià Ciurana Lanau, Harleen Hanspal, Lusine Abrahamyan, Emilio Tylson Baixauli, Yu Cao, Danai Triantafyllidou, Stefano Zorzi, Gustavo Zanoni Felipe, Peter Griger, Stefan Sietzen, Petru-Daniel TudosiuarXiv 2026 · arXiv

TL;DR

ProtoDiffusion introduces a highly efficient "token-per-layer" representation for generating and decomposing multi-layer images like graphic designs. By sharing a single spatial prototype across all layers and using individual vectors for layer-specific details, it achieves high-fidelity results while running up to 19x faster than traditional cascaded generation-and-decomposition methods.

What Problem Does This Paper Address?

Graphic designs are not just flat images; they are ordered stacks of RGBA layers (text, shapes, images). Having these layers separate is crucial for real-world use—it allows users to move a text box, swap a background, or reuse an icon.

Standard text-to-image models (like Stable Diffusion) produce a single flattened raster where all these elements are "entangled." While some recent "layered" models exist, they suffer from a major scaling problem: they represent every single layer with its own full-canvas spatial grid. If you have 10 layers, your memory and compute requirements grow roughly 10x. This makes generating complex designs with many elements prohibitively slow and memory-intensive. Furthermore, many existing models require you to specify the number of layers or the layout upfront, which limits creative flexibility.

Methods

The authors propose a two-stage framework consisting of ProtoVAE and ProtoDiffusion.

1. ProtoVAE: The Token-per-Layer Compressor

Instead of giving every layer a full grid of latent tokens, ProtoVAE factorizes a design into:

  • A Shared Prototype ($p$): A single spatial grid ($HW$ tokens) that captures the global context and shared features of the entire design.
  • Candidate Vectors ($c_i$): Exactly one small vector per layer ($N$ tokens total for $N$ layers). Each vector acts as a "key" that tells a shared decoder how to extract its specific layer from the prototype.

This reduces the representation size from $N \times HW$ tokens to just $HW + N$. As shown in Figure 2, the decoder uses these candidate vectors to modulate the shared prototype (via AdaLN-Zero) to reconstruct individual RGBA layers.

The ProtoVAE architecture showing the shared prototype and per-layer candidate extraction.

2. ProtoDiffusion: Joint Denoising

ProtoDiffusion is a latent diffusion model trained in this compact space. It uses a two-stream transformer architecture:

  • One stream processes the spatial prototype grid.
  • A second stream processes a fixed set of "candidate slots."
  • Bidirectional cross-attention allows the two streams to synchronize, ensuring that the generated layers make sense within the overall composition.

Crucially, the model determines the number of layers automatically through "activity-based selection"—it simply leaves unneeded candidate slots empty.

The ProtoDiffusion training and inference pipeline.

Datasets

The model was trained on a massive mix of layered graphic designs:

  • PrismLayersPro: A dataset of high-quality multi-layer compositions.
  • COYO-700M: Used for synthetic compositions.
  • RevealLayer: Data focused on natural-image decomposition.
  • Crello: Used as the primary benchmark for evaluation, specifically following the LayerD evaluation split.

Key Insights and Ideas

  • The Efficiency "Aha" Moment: You don't need a full latent grid to describe a single layer if you already have a global prototype. A single vector is enough to "index" into the shared spatial context and recover layer-specific identity.
  • Decoupling Content from Count: By using a fixed capacity of candidate slots (e.g., 50) and training the model to leave unused ones empty, the authors removed the need for the user to provide a "layer count" as input.
  • Unified Architecture: The same model handles three distinct tasks just by changing the conditioning: generating a design from text, decomposing an image into layers without guidance, and decomposing an image using bounding boxes for help.

Key Results

  • Speed and Scaling: ProtoDiffusion's denoising latency is nearly constant regardless of the number of layers. As shown in Figure 6, while competing models like QIL see costs explode as layers are added, ProtoDiffusion remains highly efficient.
  • Performance: It runs 19.1x faster than a comparable "generate-then-decompose" pipeline.
  • Fidelity: ProtoVAE achieves a high reconstruction quality of 40.43 dB PSNR on layer crops.
  • Generation Quality: In text-to-design tasks, it achieved an FID of 66.41, significantly outperforming a baseline using FLUX.2 (86.84) followed by a separate decomposition step.
Comparison of denoising latency and memory usage across different models.

Limitations and Future Work

  • Dense Overlaps: The model still struggles with very dense compositions where many elements overlap heavily, which can lead to "leakage" between layers.
  • Thin Elements: Preserving extremely thin graphic elements (like fine lines or small serifs in text) remains a challenge for the VAE's compression.
  • Decoding Bottleneck: While the denoising step is fast and constant-time, the final decoding of latents into pixels still scales linearly with the number of layers.
  • Natural Images: While it works on designs, the authors suggest that broader evaluation is needed to see how well this "token-per-layer" approach translates to complex natural scene decomposition (e.g., separating every leaf on a tree).

Summary updated

Abstract

Graphic designs are ordered compositions of RGBA layers whose separation enables selection, movement, and reuse. Existing layered models represent each layer with a full-canvas or region-sized spatial latent grid, so representation size grows with the cumulative spatial extent of the layers. We present ProtoDiffusion, a latent-diffusion framework built on ProtoVAE, a multi-layer variational autoencoder that encodes a design into one shared spatial prototype and one candidate vector per layer. Each candidate conditions a shared decoder to reconstruct its corresponding RGBA layer. For $N$ layers, the representation contains $HW+N$ tokens before backbone-specific packing, compared with $NHW$ for full-grid-per-layer representations. ProtoDiffusion jointly denoises the prototype and a fixed set of candidate slots, conditioned on an image or text, with bounding boxes as optional layout guidance. The model leaves unneeded slots empty and determines the number of layers and, in absence of boxes, their layout, requiring neither as input. The same architecture supports both image-to-design and text-to-design. ProtoVAE achieves high-fidelity reconstruction from its compact shared latent representation, reaching 40.43 dB PSNR on layer crops. On the LayerD evaluation split, image-to-design achieves RGB L1 scores of 0.040 with bounding boxes and 0.057 without, with box-guided alpha soft IoU (0.850) comparable to the strongest fine-tuned baseline (0.871). On merged images, text-to-design reaches 66.41 FID versus 86.84 for FLUX.2 [klein], achieves a 0.860 text-accuracy score, and runs $19.1\times$ faster than the same-backbone generation-then-decomposition cascade.
BibTeX
@misc{lanau2026protodiffusion,
  author = {Adrià Ciurana Lanau and Harleen Hanspal and Lusine Abrahamyan and Emilio Tylson Baixauli and Yu Cao and Danai Triantafyllidou and Stefano Zorzi and Gustavo Zanoni Felipe and Peter Griger and Stefan Sietzen and Petru-Daniel Tudosiu},
  title = {{ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images}},
  year = {2026},
  eprint = {2611.26586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26586},
}
Figure