Readout Guidance: Learning Control from Diffusion Features

Grace Luo, Trevor Darrell, Oliver Wang, Dan B. Goldman, Aleksander Holynski

Abstract

We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control.

Introduction

While text-to-image diffusion models have reached impressive levels of photorealism, providing users with fine-grained control over the output remains a significant challenge. Current solutions typically rely on massive architectural adapters like ControlNet, which require training auxiliary networks on hundreds of thousands of annotated images. Alternatively, sampling-time guidance methods often use heavy, external models that are not optimized for the noisy intermediate states of the diffusion process.

Readout Guidance addresses these limitations by leveraging the rich internal representations already present within frozen diffusion models. By treating these features as a latent source of geometric and semantic information, we can steer the generation process using extremely lightweight "lenses."

Methodology: Readout Heads and Guidance

The core of our approach is the Readout Head: a tiny network (often just a few convolutional or linear layers) attached to the decoder layers of a frozen U-Net. These heads are trained to "read out" specific properties—such as human pose, depth maps, or semantic correspondences—from the model's internal features.

Overview of Readout Guidance architecture and applications, showing how readout heads extract pose, appearance, and correspondence to guide the sampling process.

Guidance via Regression

Unlike traditional classifier guidance that predicts discrete categories, Readout Guidance performs regression. During the sampling process, the system calculates the distance between the current readout estimate and a user-defined target. The gradient of this distance is then backpropagated through the readout head to "nudge" the latent image at every timestep.

Relative and Non-Spatially Aligned Constraints

A key innovation of this framework is its ability to handle "relative" properties. By comparing the readout features of two different images, the model can enforce appearance similarity or point-to-point correspondence. This enables complex tasks like "drag-based" editing or identity-consistent generation without requiring per-image fine-tuning.

Data Efficiency and Performance

Because Readout Guidance bootstraps the features already learned by the diffusion model, it is remarkably data-efficient. While state-of-the-art adapters like T2I-Adapter often require millions of images, a pose readout head can be trained effectively with as few as 100 samples.

In quantitative benchmarks on SDXL, Readout Guidance trained on only 8.5k images performed comparably to a T2I-Adapter trained on 3 million images. Furthermore, the method is highly parameter-efficient; a typical readout head for SDXL is approximately 5.9M parameters, compared to 1.4GB for a full ControlNet.

A high-fidelity generation result showing precise identity and pose control achieved through Readout Guidance.

Applications

We demonstrate the versatility of Readout Guidance across several domains:

  • Spatially Aligned Control: Precise steering of pose, depth, and edges.
  • Drag-based Manipulation: Achieving state-of-the-art results on real images by manipulating semantic correspondences.
  • Identity Consistency: Maintaining character features across different generated scenes by enforcing appearance similarity readouts.
  • Refinement: Readout Guidance can act as a "last-mile" refiner on top of existing adapters to fix minor artifacts in limb placement or edge alignment.

Limitations and Future Work

The primary trade-off for this efficiency is sampling speed. Because the method requires a backward pass through the readout head at every step, generation time increases (e.g., from ~10s to ~30s for SDv1-5). Additionally, extremely high guidance weights can sometimes lead to surreal or "cartoonish" artifacts if the constraint conflicts with the model's natural image manifold. Future work will focus on optimizing the computational overhead through feature caching and improved batching strategies.

Summary updated

Abstract

We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io.
BibTeX
@misc{luo2023readout,
  author = {Grace Luo and Trevor Darrell and Oliver Wang and Dan B. Goldman and Aleksander Holynski},
  title = {{Readout Guidance: Learning Control from Diffusion Features}},
  year = {2023},
  eprint = {2312.02150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.02150},
}
Figure