TL;DR
Skip-and-Play (SnP) is a training-free method that allows diffusion models to use depth maps for precise pose control across any object category without inheriting the source object's specific shape. By selectively "skipping" specific components of the ControlNet architecture during inference, SnP enables users to generate, for example, a hedgehog in the exact pose of a horse, maintaining the hedgehog's natural silhouette.
What Problem Does This Paper Address?
Controlling the "pose" of a generated object (its orientation and position in 3D space) is a long-standing challenge in text-to-image synthesis. Current methods usually rely on two things: camera parameters (which require scarce 3D training data) or keypoints (like skeletal joints). Keypoints work well for humans but are incredibly difficult to define for arbitrary objects like a church, a car, or a hedgehog.
Depth maps are a tempting alternative because they are "category-agnostic"—you can extract a depth map from any image using off-the-shelf models. However, depth-conditional models (like standard ControlNet-Depth) suffer from shape dependency. If you use the depth map of a horse to guide the generation of a pig, you often get a "pig-shaped horse." The model struggles to distinguish between the pose (the way the animal is standing) and the shape (the long neck and slender legs of the horse).
Methods
The authors introduce Skip-and-Play (SnP), a strategy designed to extract the "pose" signal from a depth map while discarding the "shape" signal. Crucially, this is done entirely at inference time—no additional training is required.
The method is based on three key technical insights:
- Time-step Skipping: Shape is largely determined in the early stages of the diffusion denoising process. By halting the influence of ControlNet after a certain threshold (e.g., 40% of the way through), the model can use the prompt to define the object's fine-grained shape.
- Negative Prompt Feature Skipping: In standard ControlNet, features are calculated for both the positive and negative prompts. The authors discovered that removing the ControlNet features associated with the negative prompt actually improves pose accuracy while reducing shape leakage.
- Selective Decoder Blocks: Not all layers in a neural network are created equal. Through ablation, the authors found that the Middle Block (MB) and the fourth decoder block of the U-Net are the primary drivers of pose. By only applying ControlNet features to these specific blocks, they preserve the pose while letting other layers handle the object's identity.
Additionally, they propose an optional Weight Map Control Module (WCM) that uses an edge detector to identify and down-weight the boundaries of the depth map, further preventing the model from "tracing" the source object's silhouette.
Datasets
The researchers evaluated SnP using several datasets to prove its versatility:
- FFHQ: A standard dataset of human faces used to measure pose accuracy (rotation/elevation).
- PoseH: A new benchmark constructed by the authors. It consists of images rendered from a single 3D mesh with a uniform distribution of poses to evaluate how well the model handles extreme or non-frontal angles.
- Stanford Cars & LSUN Church: Used to demonstrate the "Any Object" capability of the model.
Key Insights and Ideas
- The "Where" vs. "What" of Layers: The most significant insight is the empirical mapping of ControlNet layers to image attributes. Finding that the Middle Block handles coarse pose while later blocks handle fine shape allows for surgical intervention in the generation process.
- The Negative Prompt Paradox: Intuitively, one might think more information (positive + negative features) leads to better control. However, the paper shows that the negative prompt's ControlNet features often introduce conflicting shape constraints that actually hinder the "joint" application of the pose and the text prompt.
- Depth as a Universal Proxy: The paper reinforces that depth is a superior control signal to 2D keypoints because it inherently encodes 3D spatial information, making it much more robust for side views and complex orientations.
Key Results
- Superior Pose Accuracy: On the FFHQ and PoseH datasets, SnP achieved significantly lower pose errors than OpenPose-based ControlNet, especially as the pose moved away from a simple frontal view.
- Better Prompt Adherence: Quantitatively, SnP showed higher CLIP similarity scores compared to standard ControlNet-Depth. This means the final image looked more like the "text prompt" (e.g., a pig) and less like the "condition" (e.g., a horse).
- Qualitative Versatility: The paper showcases impressive "cross-category" generations, such as turning a leopard's depth map into a sloth or a pig while perfectly preserving the head tilt and body orientation of the original leopard.
Limitations and Future Work
The authors acknowledge that SnP is a "training-free" method that leverages the prior knowledge already inside models like Stable Diffusion. This means if the base model hasn't seen enough examples of a specific object in a specific pose (e.g., a very rare angle of a specific animal), SnP cannot magically create it. The performance is ultimately bounded by the capabilities of the underlying diffusion model and the quality of the initial ControlNet.
Future work could involve fine-tuning these "skipping" parameters automatically based on the complexity of the input depth map.
Summary updated