SpotActor: Training-Free Layout-Controlled Consistent Image Generation

Jiahao Wang, Caixia Yan, Weizhan Zhang, Haonan Lin, Mengmeng Wang, Guang Dai, Tieliang Gong, Hao Sun, Jingdong Wang

Introduction

Generating a series of images where specific characters maintain a consistent appearance while appearing in precise, user-defined locations is a significant challenge in the field of text-to-image (T2I) diffusion. This task, termed "Layout-to-Consistent-Image" (L2CI), is essential for creative workflows such as comic book production or storyboarding. While standard T2I models are powerful, they often struggle to balance spatial layout control with subject consistency across multiple frames.

Existing methods typically address these requirements in isolation. Layout-control techniques often overlook character identity, while consistency-focused models frequently fail to adhere to strict spatial constraints. Furthermore, many layout methods optimize only the latent noise, neglecting the semantic text representation. SpotActor addresses these gaps with a training-free framework that optimizes both latent and semantic spaces to achieve superior alignment and consistency.

Methodology: Dual Space Optimization

SpotActor introduces a two-stage pipeline that operates without the need for fine-tuning. The core innovation lies in its "Dual Energy Guidance" system. Unlike previous approaches that only update latent codes, SpotActor treats the latent and semantic spaces as a unified "dual space." By performing backward updates on both the latent noise and the text embeddings, the model achieves a more optimal visual-textual alignment.

Nuanced Layout Energy Function

A key finding of the authors is that standard "hard" bounding box constraints can be detrimental, often leading to visual artifacts or subjects that do not naturally fill the intended space. To solve this, SpotActor employs a 2D sigmoid-based objective function. This function mimics the natural peak-and-fade activation patterns found in a U-Net's attention maps, allowing the model to place subjects more fluidly within the specified layout.

Consistent Forward Sampling

During the image generation phase, SpotActor utilizes two novel attention mechanisms to synchronize character appearance across a batch:

  1. Regional Interconnection Self-Attention (RISA): This mechanism allows spatial features to interact across different images in a batch, specifically within the regions defined by the layout.
  2. Semantic Fusion Cross-Attention (SFCA): This enables images to share semantic information about the subject, ensuring that fine details like clothing texture or hair color remain identical across the series.
A fairy in a silver dress and a unicorn walking on a street, demonstrating precise layout control and high-quality rendering.

ActorBench and Performance

To evaluate the L2CI task, the authors introduced ActorBench, a comprehensive benchmark containing 200 sets of prompt-box pairs (100 single-subject and 100 double-subject). The layouts are derived from real-world object detection data (COCO2017), and prompts are generated via ChatGPT to ensure diverse scenarios while maintaining constant subject descriptions.

In comparative studies against state-of-the-art baselines like StoryDiffusion and LayoutGuidance, SpotActor demonstrated significant improvements:

  • Layout Alignment: Achieved a 67.1% mIoU, compared to 53.7% for LayoutGuidance.
  • Subject Consistency: Recorded the highest scores in visual similarity metrics (78.6% DINO-fg and 79.7% CLIP-fg).
The same fairy and unicorn characters running in a park, showcasing the model's ability to maintain character consistency across different backgrounds and actions.

Discussion and Limitations

The success of SpotActor highlights the importance of the semantic space in layout-controlled generation. By tweaking the "meaning" of the prompt alongside the image noise, the model finds a better "spot" for the pixels. However, there are trade-offs. Like many constrained generation methods, there is a slight reduction in overall prompt conformity (CLIP-T score) as the model prioritizes layout and consistency. Additionally, the optimization stage involves backward passes at each sampling step, making it computationally more intensive than standard single-pass generation, though it remains more efficient than methods requiring per-subject fine-tuning.

Conclusion

SpotActor provides a robust, training-free solution for the L2CI task. By combining dual-space optimization with nuanced energy functions and cross-image attention mechanisms, it enables the creation of consistent visual narratives with precise spatial control. Future work may focus on improving background diversity and reducing inference overhead to make the framework even more accessible for real-time creative applications.

Summary updated

Abstract

Text-to-image diffusion models significantly enhance the efficiency of artistic creation with high-fidelity image generation. However, in typical application scenarios like comic book production, they can neither place each subject into its expected spot nor maintain the consistent appearance of each subject across images. For these issues, we pioneer a novel task, Layout-to-Consistent-Image (L2CI) generation, which produces consistent and compositional images in accordance with the given layout conditions and text prompts. To accomplish this challenging task, we present a new formalization of dual energy guidance with optimization in a dual semantic-latent space and thus propose a training-free pipeline, SpotActor, which features a layout-conditioned optimizing stage and a consistent sampling stage. In the optimizing stage, we innovate a nuanced layout energy function to mimic the attention activations with a sigmoid-like objective. While in the sampling stage, we design Regional Interconnection Self-Attention (RISA) and Semantic Fusion Cross-Attention (SFCA) mechanisms that allow mutual interactions across images. To evaluate the performance, we present ActorBench, a specified benchmark with hundreds of reasonable prompt-box pairs stemming from object detection datasets. Comprehensive experiments are conducted to demonstrate the effectiveness of our method. The results prove that SpotActor fulfills the expectations of this task and showcases the potential for practical applications with superior layout alignment, subject consistency, prompt conformity and background diversity.
BibTeX
@inproceedings{wang2025spotactor,
  author = {Jiahao Wang and Caixia Yan and Weizhan Zhang and Haonan Lin and Mengmeng Wang and Guang Dai and Tieliang Gong and Hao Sun and Jingdong Wang},
  title = {{SpotActor: Training-Free Layout-Controlled Consistent Image Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32831},
}
Figure