OneActor: Consistent Character Generation via Cluster-Conditioned Guidance

Jiahao Wang, Caixia Yan, Haonan Lin, Weizhan Zhang

Abstract

OneActor is a novel "one-shot" tuning paradigm designed to enable text-to-image (T2I) diffusion models to generate consistent characters or objects across diverse prompts. By framing the consistency challenge as a navigation problem within the latent space's existing clusters, OneActor achieves superior subject fidelity and prompt adherence. The method offers 4x faster tuning than existing state-of-the-art techniques with zero inference overhead, providing a robust solution for identity-consistent image generation.

1. Introduction

Standard text-to-image (T2I) models, such as Stable Diffusion, often exhibit high stochasticity, making it difficult to maintain character identity across different prompts. For instance, requesting a "hobbit with robes" followed by a "hobbit running" typically results in two distinct individuals. Existing solutions generally fall into two categories: personalization methods like DreamBooth, which require multiple images and expensive fine-tuning, or tuning-free mechanisms like ConsiStory, which often increase inference time or struggle with abstract subjects.

OneActor identifies a middle ground by leveraging the observation that a pre-trained model's latent space already contains "clusters" of similar-looking subjects. The primary challenge is not teaching the model a new character, but "taming" it to remain within a specific identity sub-cluster while following new textual instructions.

Comparison between ordinary T2I models and OneActor's cluster-guided denoising trajectory.

2. Methodology

OneActor introduces a cluster-conditioned guidance approach, utilizing a single "target" image to guide subsequent generations.

2.1 The Projector Architecture

Rather than fine-tuning the entire U-Net backbone, OneActor employs a lightweight Projector network consisting of ResNet, Linear, and AdaIN layers. This projector processes the target image features and the prompt's text embedding to generate a "semantic offset" ($\Delta c$). This offset nudges the model toward the specific identity of the target without sacrificing prompt flexibility.

2.2 One-Shot Tuning with Auxiliary Samples

To avoid overfitting to a single pose or background, OneActor utilizes auxiliary samples during a brief 3–6 minute tuning phase. The training objective involves a "push-pull" mechanism:

  • Attraction: The denoising trajectory is pulled toward the target image's identity.
  • Repulsion: The trajectory is pushed away from "auxiliary" images (rejected versions of the subject). This ensures the model captures the unique nuances of the selected identity.
Overview of the OneActor framework, including data preparation, tuning, and the internal projector architecture.

2.3 Inference and Guidance

The authors demonstrate a "semantic-latent guidance equivalence," allowing for fine-grained control via text embedding manipulation. During generation, a modified score function incorporates target attraction and auxiliary exclusion terms to maintain consistency.

3. Evaluation and Results

OneActor was evaluated using SDXL as a backbone across diverse subject categories (humans, animals, objects) and ChatGPT-generated prompt templates.

  • Efficiency: Tuning takes only 3–6 minutes on a single A100 GPU, significantly faster than baselines like TheChosenOne.
  • Inference Performance: By adjusting guidance factors, OneActor can achieve zero inference overhead compared to standard SDXL.
  • Subject Fidelity: Quantitative metrics (CLIP and DINO scores) show that OneActor outperforms IP-Adapter and DreamBooth in balancing subject consistency and prompt similarity.
  • Qualitative Quality: The model excels at preserving fine details, such as clothing color and object textures, across varied environments.

4. Discussion and Limitations

The core insight of OneActor is viewing identity as a sub-cluster navigation problem. By emphasizing repulsion from non-target samples, the model effectively isolates desired identities.

However, limitations remain:

  • Subject Density: Performance degrades when generating more than four consistent subjects in one image.
  • Latent Entanglement: Specific objects (e.g., an umbrella) can become "entangled" with the character's identity.
  • OOD Prompts: Guidance may fail if the base cluster lacks data for specific requested actions.

Future work may involve pre-training a universal network using this cluster-conditioned logic to eliminate the per-subject tuning phase entirely.

Summary updated

Abstract

Text-to-image diffusion models benefit artists with high-quality image generation. Yet their stochastic nature hinders artists from creating consistent images of the same subject. Existing methods try to tackle this challenge and generate consistent content in various ways. However, they either depend on external restricted data or require expensive tuning of the diffusion model. For this issue, we propose a novel one-shot tuning paradigm, termed as OneActor. It efficiently performs consistent subject generation solely driven by prompts via a learned semantic guidance to bypass the laborious backbone tuning. We lead the way to formalize the objective of consistent subject generation from a clustering perspective, and thus design a cluster-conditioned model. To mitigate the overfitting challenge shared by one-shot tuning pipelines, we augment the tuning with auxiliary samples and devise two inference strategies: semantic interpolation and cluster guidance. These techniques are later verified to significantly enhance the generation quality. Comprehensive experiments show that our method outperforms a variety of baselines with satisfactory subject consistency, superior prompt conformity as well as high image quality. Our method is capable of multi-subject generation and compatible with popular diffusion extensions. Besides, we achieve a 4 times faster tuning speed than tuning-based baselines and, if desired, avoid increasing inference time. Furthermore, to our best knowledge, we are the first to prove that the semantic space of the diffusion model has the same interpolation property as the latent space does. This property can serve as another promising tool for fine generation control.
BibTeX
@inproceedings{wang2024oneactor,
  author = {Jiahao Wang and Caixia Yan and Haonan Lin and Weizhan Zhang},
  title = {{OneActor: Consistent Character Generation via Cluster-Conditioned Guidance}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=2gtNa14V45},
}
Figure