Abstract
OneActor is a novel "one-shot" tuning paradigm designed to enable text-to-image (T2I) diffusion models to generate consistent characters or objects across diverse prompts. By framing the consistency challenge as a navigation problem within the latent space's existing clusters, OneActor achieves superior subject fidelity and prompt adherence. The method offers 4x faster tuning than existing state-of-the-art techniques with zero inference overhead, providing a robust solution for identity-consistent image generation.
1. Introduction
Standard text-to-image (T2I) models, such as Stable Diffusion, often exhibit high stochasticity, making it difficult to maintain character identity across different prompts. For instance, requesting a "hobbit with robes" followed by a "hobbit running" typically results in two distinct individuals. Existing solutions generally fall into two categories: personalization methods like DreamBooth, which require multiple images and expensive fine-tuning, or tuning-free mechanisms like ConsiStory, which often increase inference time or struggle with abstract subjects.
OneActor identifies a middle ground by leveraging the observation that a pre-trained model's latent space already contains "clusters" of similar-looking subjects. The primary challenge is not teaching the model a new character, but "taming" it to remain within a specific identity sub-cluster while following new textual instructions.
2. Methodology
OneActor introduces a cluster-conditioned guidance approach, utilizing a single "target" image to guide subsequent generations.
2.1 The Projector Architecture
Rather than fine-tuning the entire U-Net backbone, OneActor employs a lightweight Projector network consisting of ResNet, Linear, and AdaIN layers. This projector processes the target image features and the prompt's text embedding to generate a "semantic offset" ($\Delta c$). This offset nudges the model toward the specific identity of the target without sacrificing prompt flexibility.
2.2 One-Shot Tuning with Auxiliary Samples
To avoid overfitting to a single pose or background, OneActor utilizes auxiliary samples during a brief 3–6 minute tuning phase. The training objective involves a "push-pull" mechanism:
- Attraction: The denoising trajectory is pulled toward the target image's identity.
- Repulsion: The trajectory is pushed away from "auxiliary" images (rejected versions of the subject). This ensures the model captures the unique nuances of the selected identity.
2.3 Inference and Guidance
The authors demonstrate a "semantic-latent guidance equivalence," allowing for fine-grained control via text embedding manipulation. During generation, a modified score function incorporates target attraction and auxiliary exclusion terms to maintain consistency.
3. Evaluation and Results
OneActor was evaluated using SDXL as a backbone across diverse subject categories (humans, animals, objects) and ChatGPT-generated prompt templates.
- Efficiency: Tuning takes only 3–6 minutes on a single A100 GPU, significantly faster than baselines like TheChosenOne.
- Inference Performance: By adjusting guidance factors, OneActor can achieve zero inference overhead compared to standard SDXL.
- Subject Fidelity: Quantitative metrics (CLIP and DINO scores) show that OneActor outperforms IP-Adapter and DreamBooth in balancing subject consistency and prompt similarity.
- Qualitative Quality: The model excels at preserving fine details, such as clothing color and object textures, across varied environments.
4. Discussion and Limitations
The core insight of OneActor is viewing identity as a sub-cluster navigation problem. By emphasizing repulsion from non-target samples, the model effectively isolates desired identities.
However, limitations remain:
- Subject Density: Performance degrades when generating more than four consistent subjects in one image.
- Latent Entanglement: Specific objects (e.g., an umbrella) can become "entangled" with the character's identity.
- OOD Prompts: Guidance may fail if the base cluster lacks data for specific requested actions.
Future work may involve pre-training a universal network using this cluster-conditioned logic to eliminate the per-subject tuning phase entirely.
Summary updated