Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Objects

Kyungmin Jo, Jaegul Choo

TL;DR

Skip-and-Play (SnP) is a training-free method that allows diffusion models to use depth maps for precise pose control across any object category without inheriting the source object's specific shape. By selectively "skipping" specific components of the ControlNet architecture during inference, SnP enables users to generate, for example, a hedgehog in the exact pose of a horse, maintaining the hedgehog's natural silhouette.

What Problem Does This Paper Address?

Controlling the "pose" of a generated object (its orientation and position in 3D space) is a long-standing challenge in text-to-image synthesis. Current methods usually rely on two things: camera parameters (which require scarce 3D training data) or keypoints (like skeletal joints). Keypoints work well for humans but are incredibly difficult to define for arbitrary objects like a church, a car, or a hedgehog.

Depth maps are a tempting alternative because they are "category-agnostic"—you can extract a depth map from any image using off-the-shelf models. However, depth-conditional models (like standard ControlNet-Depth) suffer from shape dependency. If you use the depth map of a horse to guide the generation of a pig, you often get a "pig-shaped horse." The model struggles to distinguish between the pose (the way the animal is standing) and the shape (the long neck and slender legs of the horse).

Methods

The authors introduce Skip-and-Play (SnP), a strategy designed to extract the "pose" signal from a depth map while discarding the "shape" signal. Crucially, this is done entirely at inference time—no additional training is required.

The method is based on three key technical insights:

  1. Time-step Skipping: Shape is largely determined in the early stages of the diffusion denoising process. By halting the influence of ControlNet after a certain threshold (e.g., 40% of the way through), the model can use the prompt to define the object's fine-grained shape.
  2. Negative Prompt Feature Skipping: In standard ControlNet, features are calculated for both the positive and negative prompts. The authors discovered that removing the ControlNet features associated with the negative prompt actually improves pose accuracy while reducing shape leakage.
  3. Selective Decoder Blocks: Not all layers in a neural network are created equal. Through ablation, the authors found that the Middle Block (MB) and the fourth decoder block of the U-Net are the primary drivers of pose. By only applying ControlNet features to these specific blocks, they preserve the pose while letting other layers handle the object's identity.

Additionally, they propose an optional Weight Map Control Module (WCM) that uses an edge detector to identify and down-weight the boundaries of the depth map, further preventing the model from "tracing" the source object's silhouette.

Datasets

The researchers evaluated SnP using several datasets to prove its versatility:

  • FFHQ: A standard dataset of human faces used to measure pose accuracy (rotation/elevation).
  • PoseH: A new benchmark constructed by the authors. It consists of images rendered from a single 3D mesh with a uniform distribution of poses to evaluate how well the model handles extreme or non-frontal angles.
  • Stanford Cars & LSUN Church: Used to demonstrate the "Any Object" capability of the model.

Key Insights and Ideas

  • The "Where" vs. "What" of Layers: The most significant insight is the empirical mapping of ControlNet layers to image attributes. Finding that the Middle Block handles coarse pose while later blocks handle fine shape allows for surgical intervention in the generation process.
  • The Negative Prompt Paradox: Intuitively, one might think more information (positive + negative features) leads to better control. However, the paper shows that the negative prompt's ControlNet features often introduce conflicting shape constraints that actually hinder the "joint" application of the pose and the text prompt.
  • Depth as a Universal Proxy: The paper reinforces that depth is a superior control signal to 2D keypoints because it inherently encodes 3D spatial information, making it much more robust for side views and complex orientations.

Key Results

  • Superior Pose Accuracy: On the FFHQ and PoseH datasets, SnP achieved significantly lower pose errors than OpenPose-based ControlNet, especially as the pose moved away from a simple frontal view.
  • Better Prompt Adherence: Quantitatively, SnP showed higher CLIP similarity scores compared to standard ControlNet-Depth. This means the final image looked more like the "text prompt" (e.g., a pig) and less like the "condition" (e.g., a horse).
  • Qualitative Versatility: The paper showcases impressive "cross-category" generations, such as turning a leopard's depth map into a sloth or a pig while perfectly preserving the head tilt and body orientation of the original leopard.

Limitations and Future Work

The authors acknowledge that SnP is a "training-free" method that leverages the prior knowledge already inside models like Stable Diffusion. This means if the base model hasn't seen enough examples of a specific object in a specific pose (e.g., a very rare angle of a specific animal), SnP cannot magically create it. The performance is ultimately bounded by the capabilities of the underlying diffusion model and the quality of the initial ControlNet.

Future work could involve fine-tuning these "skipping" parameters automatically based on the complexity of the input depth map.

Summary updated

Abstract

The emergence of diffusion models has enabled the generation of diverse high-quality images solely from text, prompting subsequent efforts to enhance the controllability of these models. Despite the improvement in controllability, pose control remains limited to specific objects (e.g., humans) or poses (e.g., frontal view) due to the fact that pose is generally controlled via camera parameters (e.g., rotation angle) or keypoints (e.g., eyes, nose). Specifically, camera parameters-conditional pose control models generate unrealistic images depending on the object, owing to the small size of 3D datasets for training. Also, keypoint-based approaches encounter challenges in acquiring reliable keypoints for various objects (e.g., church) or poses (e.g., back view). To address these limitations, we propose depth-based pose control, as depth maps are easily obtainable from a single depth estimation model regardless of objects and poses, unlike camera parameters and keypoints. However, depth-based pose control confronts issues of shape dependency, as depth maps influence not only the pose but also the shape of the generated images. To tackle this issue, we propose Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional ControlNet on the pose and the shape of the generated images. To be specific, based on the analysis, we selectively skip parts of the components to mitigate shape dependency on the depth map while preserving the pose. Through various experiments, we demonstrate the superiority of SnP over baselines and showcase the ability of SnP to generate images of diverse objects and poses. Remarkably, SnP exhibits the ability to generate images even when the objects in the condition (e.g., a horse) and the prompt (e.g., a hedgehog) differ from each other.
BibTeX
@inproceedings{jo2025skip,
  author = {Kyungmin Jo and Jaegul Choo},
  title = {{Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Objects}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00077},
}
Figure