Abstract
Skip-and-Play (SnP) is a training-free inference strategy that enables diffusion models to utilize depth maps for precise pose control across any object category without inheriting the source object's specific shape. By selectively skipping components of the ControlNet architecture during inference, SnP allows for generating diverse objects—such as a hedgehog or a pig—in the exact pose of a reference horse, maintaining the target object's natural silhouette.
Introduction: The Challenge of Shape-Pose Decoupling
Controlling the "pose" of a generated object—its orientation and position in 3D space—is a fundamental challenge in text-to-image synthesis. While camera parameters or keypoints are often used, they require scarce 3D data or are difficult to define for arbitrary objects like buildings or animals. Depth maps offer a category-agnostic alternative, but standard depth-conditional models (like ControlNet-Depth) suffer from shape dependency. When using a horse's depth map to guide the generation of a pig, the result is often a "pig-shaped horse" because the model fails to distinguish between the pose (the stance) and the shape (the horse's specific anatomy).
Methodology: Selective Feature Skipping
The authors introduce Skip-and-Play (SnP), a strategy to extract the pose signal from a depth map while discarding the shape signal. This is achieved entirely at inference time without additional training through three key insights:
- Time-step Skipping: Since shape is primarily determined in the early stages of denoising, halting ControlNet influence after approximately 40% of the process allows the text prompt to define the object's fine-grained shape.
- Negative Prompt Feature Skipping: Removing ControlNet features associated with the negative prompt improves pose accuracy and reduces shape leakage, as these features often introduce conflicting shape constraints.
- Selective Decoder Blocks: Empirical analysis revealed that the Middle Block (MB) and the fourth decoder block of the U-Net are the primary drivers of pose. By applying ControlNet features only to these blocks, SnP preserves pose while letting other layers handle object identity.
Additionally, an optional Weight Map Control Module (WCM) uses edge detection to down-weight depth map boundaries, further preventing the model from tracing the source object's silhouette.
Key Results and Insights
SnP was evaluated on datasets including FFHQ (faces), Stanford Cars, and a new benchmark, PoseH, which uses 3D meshes to test extreme angles.
- Superior Pose Accuracy: SnP achieved significantly lower pose errors than OpenPose-based ControlNet, particularly for non-frontal views.
- Enhanced Prompt Adherence: Quantitatively, SnP showed higher CLIP similarity scores than standard ControlNet-Depth, indicating that the generated images aligned better with the text prompt (e.g., "a pig") than the depth condition (e.g., "a horse").
- Cross-Category Versatility: The method successfully generates diverse animals in the pose of a reference object without inheriting its silhouette.
Conclusion
Skip-and-Play demonstrates that the "where" and "when" of control signals in diffusion models are critical. By surgically intervening in the ControlNet architecture, SnP provides a robust, training-free solution for universal pose control. While performance remains bounded by the base model's prior knowledge, SnP significantly advances the ability to decouple spatial orientation from object identity.
Summary updated