Abstract
While diffusion models have revolutionized image generation, the internal mechanisms by which they transform Gaussian noise into coherent scenes remain largely opaque. This research utilizes Sparse Autoencoders (SAEs) to decompose the internal activations of Stable Diffusion, revealing that high-level scene composition is established during the very first denoising step. By mapping these internal features to human-interpretable concepts, the authors identify distinct temporal windows for controlling layout, style, and texture, providing a new framework for mechanistic interpretability in generative AI.
The Mystery of the First Step
Researchers have long sought to identify the exact moment a diffusion model "decides" the final composition of an image. Traditionally, the early stages of diffusion—where the output appears as unrecognizable pixel noise—were thought to be purely chaotic. However, by applying SAEs to the cross-attention layers of the U-Net, this study demonstrates that the model establishes the scene's layout at $t=1.0$, the very first step of the process.
As shown in the figure above, even when the pixel-space output is indistinguishable from random static, the internal "concepts" captured by the SAE can predict the final segmentation masks of objects like palm trees and people. This suggests that the model's "plan" for the image is formed almost instantaneously.
Methodology: Sparse Autoencoders for Vision
To peek inside the "black box," the authors employed Sparse Autoencoders (SAEs), a technique originally developed to interpret Large Language Models (LLMs). SAEs decompose complex neural activations into an "over-complete" set of sparse, interpretable features.
The Intervention Pipeline
The researchers trained SAEs on the residual updates of Stable Diffusion v1.4's cross-attention layers. To label these features without human bias, they developed an automated vision pipeline using object detectors (Grounding DINO) and segmentors (SAM). This allowed them to correlate specific SAE activations with "ground truth" visual concepts like "apple," "building," or "watercolor style."
By intervening on these features—either by boosting them globally or restricting them to specific spatial quadrants—the authors could steer the model's output in a controlled, predictable manner.
The Temporal Control Hierarchy
A key finding of the paper is that the effectiveness of steering a model is highly dependent on the diffusion timestep. The generation process follows a strict hierarchy of concept emergence:
- Early Stage ($t > 0.6$): This is the window for Layout Control. Interventions here can move objects to different parts of the frame. However, the model is resistant to style changes at this stage.
- Middle Stage ($0.2 < t < 0.6$): This is the "sweet spot" for Style Control. While the layout is now "locked in" (attempting to move objects here causes visual artifacts), the model is highly receptive to changes in artistic medium or lighting. Global style interventions at $t=0.5$ achieved a 93% success rate in increasing CLIP similarity to target styles.
- Late Stage ($t < 0.2$): The model enters a Refinement Phase. Interventions here only affect microscopic textures and fine details, leaving the overall structure and style untouched.
Results and Implications
The study's automated Concept Dictionary and layout prediction metrics confirm that semantic granularity increases over time. Early features represent generic categories (e.g., "furniture"), while later features distinguish specific components (e.g., "chair legs"). The SAEs achieved an Explained Variance of up to 77.7%, indicating they capture a substantial portion of the model's internal logic.
Despite these successes, the authors noted "leakage" where information bypasses interventions via U-Net skip connections. Future work may involve applying these techniques to Diffusion Transformers (DiTs), which may offer even more granular and cleaner control over the generative process. This research marks a significant step toward moving diffusion models from "black boxes" to steerable, interpretable tools for creative expression.
Summary updated