TL;DR
ProtoDiffusion introduces a highly efficient "token-per-layer" representation for generating and decomposing multi-layer images like graphic designs. By sharing a single spatial prototype across all layers and using individual vectors for layer-specific details, it achieves high-fidelity results while running up to 19x faster than traditional cascaded generation-and-decomposition methods.
What Problem Does This Paper Address?
Graphic designs are not just flat images; they are ordered stacks of RGBA layers (text, shapes, images). Having these layers separate is crucial for real-world use—it allows users to move a text box, swap a background, or reuse an icon.
Standard text-to-image models (like Stable Diffusion) produce a single flattened raster where all these elements are "entangled." While some recent "layered" models exist, they suffer from a major scaling problem: they represent every single layer with its own full-canvas spatial grid. If you have 10 layers, your memory and compute requirements grow roughly 10x. This makes generating complex designs with many elements prohibitively slow and memory-intensive. Furthermore, many existing models require you to specify the number of layers or the layout upfront, which limits creative flexibility.
Methods
The authors propose a two-stage framework consisting of ProtoVAE and ProtoDiffusion.
1. ProtoVAE: The Token-per-Layer Compressor
Instead of giving every layer a full grid of latent tokens, ProtoVAE factorizes a design into:
- A Shared Prototype ($p$): A single spatial grid ($HW$ tokens) that captures the global context and shared features of the entire design.
- Candidate Vectors ($c_i$): Exactly one small vector per layer ($N$ tokens total for $N$ layers). Each vector acts as a "key" that tells a shared decoder how to extract its specific layer from the prototype.
This reduces the representation size from $N \times HW$ tokens to just $HW + N$. As shown in Figure 2, the decoder uses these candidate vectors to modulate the shared prototype (via AdaLN-Zero) to reconstruct individual RGBA layers.
2. ProtoDiffusion: Joint Denoising
ProtoDiffusion is a latent diffusion model trained in this compact space. It uses a two-stream transformer architecture:
- One stream processes the spatial prototype grid.
- A second stream processes a fixed set of "candidate slots."
- Bidirectional cross-attention allows the two streams to synchronize, ensuring that the generated layers make sense within the overall composition.
Crucially, the model determines the number of layers automatically through "activity-based selection"—it simply leaves unneeded candidate slots empty.
Datasets
The model was trained on a massive mix of layered graphic designs:
- PrismLayersPro: A dataset of high-quality multi-layer compositions.
- COYO-700M: Used for synthetic compositions.
- RevealLayer: Data focused on natural-image decomposition.
- Crello: Used as the primary benchmark for evaluation, specifically following the LayerD evaluation split.
Key Insights and Ideas
- The Efficiency "Aha" Moment: You don't need a full latent grid to describe a single layer if you already have a global prototype. A single vector is enough to "index" into the shared spatial context and recover layer-specific identity.
- Decoupling Content from Count: By using a fixed capacity of candidate slots (e.g., 50) and training the model to leave unused ones empty, the authors removed the need for the user to provide a "layer count" as input.
- Unified Architecture: The same model handles three distinct tasks just by changing the conditioning: generating a design from text, decomposing an image into layers without guidance, and decomposing an image using bounding boxes for help.
Key Results
- Speed and Scaling: ProtoDiffusion's denoising latency is nearly constant regardless of the number of layers. As shown in Figure 6, while competing models like QIL see costs explode as layers are added, ProtoDiffusion remains highly efficient.
- Performance: It runs 19.1x faster than a comparable "generate-then-decompose" pipeline.
- Fidelity: ProtoVAE achieves a high reconstruction quality of 40.43 dB PSNR on layer crops.
- Generation Quality: In text-to-design tasks, it achieved an FID of 66.41, significantly outperforming a baseline using FLUX.2 (86.84) followed by a separate decomposition step.
Limitations and Future Work
- Dense Overlaps: The model still struggles with very dense compositions where many elements overlap heavily, which can lead to "leakage" between layers.
- Thin Elements: Preserving extremely thin graphic elements (like fine lines or small serifs in text) remains a challenge for the VAE's compression.
- Decoding Bottleneck: While the denoising step is fast and constant-time, the final decoding of latents into pixels still scales linearly with the number of layers.
- Natural Images: While it works on designs, the authors suggest that broader evaluation is needed to see how well this "token-per-layer" approach translates to complex natural scene decomposition (e.g., separating every leaf on a tree).
Summary updated