PALP: Prompt Aligned Personalization of Text-to-Image Models

Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, Ariel Shamir

Abstract

Personalizing text-to-image (T2I) models to represent specific subjects often results in a significant trade-off: the model either captures the subject's identity with high fidelity but ignores the prompt's context, or follows the prompt while losing the subject's unique characteristics. We present PALP (Prompt Aligned Personalization), a framework that utilizes a novel score distillation technique to maintain alignment with complex, multi-element instructions during the fine-tuning process. By regularizing the personalization process with a Delta Denoising Score (DDS), PALP enables the generation of images that are both faithful to a specific subject and strictly adherent to intricate textual prompts.

Introduction

Large-scale text-to-image models, such as Stable Diffusion, have demonstrated a remarkable ability to generate diverse imagery from natural language. However, these models lack the inherent knowledge of specific, user-provided subjects (e.g., a specific pet, a personal item, or a unique building). Existing personalization techniques like DreamBooth and Textual Inversion attempt to bridge this gap by fine-tuning models on a small set of reference images.

Despite their success, these methods frequently suffer from "identity vs. prompt" interference. When a model is heavily optimized to replicate a specific subject, it often overfits, causing it to "forget" its prior knowledge of styles, lighting, and spatial relationships. For instance, a model trained on a specific cat might generate that cat perfectly but fail to place it in a "charcoal sketch style on the moon" because the subject's features overwrite the stylistic and contextual cues of the prompt. PALP addresses this limitation, allowing for high-fidelity personalization within highly specific and complex scenes.

Methodology

PALP operates by optimizing the model through two parallel paths during the training phase, ensuring that subject learning does not come at the cost of prompt adherence.

The Personalization Path

This path follows standard personalization protocols. It utilizes Low-Rank Adaptation (LoRA) to update a targeted subset of the model's weights while simultaneously optimizing a new word embedding (a placeholder token [V]) to represent the subject. A standard reconstruction loss is applied to ensure the model learns the visual features of the reference subject.

The Prompt-Alignment Path

The core innovation of PALP is the integration of Score Distillation Sampling (SDS) as a regularizer. Specifically, the authors employ a variant known as Delta Denoising Score (DDS). During training, the framework evaluates a "clean" version of the target prompt (e.g., "A sketch of a cat" instead of "A sketch of [my cat]").

By calculating the difference (the "delta") between how the original pre-trained model denoises the clean prompt and how the personalized model denoises the subject-specific prompt, PALP forces the model to maintain the original model's understanding of style and context. This tethering mechanism prevents the subject's identity from dominating the latent space, ensuring that elements like "Paris," "Pop Art," or "Underwater" are preserved in the final output.

Experimental Evaluation

The researchers benchmarked PALP across three primary scenarios:

  • Multi-shot Personalization: Utilizing standard datasets featuring animals, toys, and architecture to test general subject fidelity.
  • Single-shot Personalization: Evaluating the model's ability to learn a subject (specifically human portraits) from only one reference image.
  • Complex Prompt Benchmark: A specialized test set consisting of ten intricate prompts with at least four distinct elements (e.g., specific styles, times of day, and color palettes) to measure the limits of prompt alignment.

Results and Discussion

PALP demonstrates significant improvements over baseline methods such as P+, NeTI, and DreamBooth in both qualitative and quantitative assessments.

  • Text Alignment: In human evaluation studies, PALP achieved a 91.2% text alignment score, significantly higher than the 73.3% achieved by the next best method (TI+DB). This indicates that PALP is uniquely capable of incorporating multiple complex prompt elements without omission.
  • Subject Fidelity: The model maintained a 72.1% personalization score, confirming that the increased prompt alignment does not result in a loss of the subject's identity.
  • Qualitative Performance: Visual analysis shows that PALP successfully renders subjects in challenging artistic styles, such as "Vitruvian Man" sketches or "Risograph" prints, where competing models typically fail to either maintain the style or preserve the subject's likeness.

Conclusion and Future Work

PALP effectively mitigates the trade-off between subject fidelity and prompt alignment by leveraging the prior knowledge of pre-trained models through score distillation. While the current implementation is optimized for a specific target prompt—making it a "per-scene" optimization tool—it sets a new standard for high-fidelity, complex AI art generation. Future research will focus on developing prompt-aligned adapters to enable "instant" personalization, removing the need for per-scene fine-tuning while maintaining the high alignment standards established by PALP.

Summary updated

Abstract

Content creators often aim to create personalized images using personal subjects that go beyond the capabilities of conventional text-to-image models. Additionally, they may want the resulting image to encompass a specific location, style, ambiance, and more. Existing personalization methods may compromise personalization ability or the alignment to complex textual prompts. This trade-off can impede the fulfillment of user prompts and subject fidelity. We propose a new approach focusing on personalization methods for a \emph{single} prompt to address this issue. We term our approach prompt-aligned personalization. While this may seem restrictive, our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts, which may pose a challenge for current techniques. In particular, our method keeps the personalized model aligned with a target prompt using an additional score distillation sampling term. We demonstrate the versatility of our method in multi- and single-shot settings and further show that it can compose multiple subjects or use inspiration from reference images, such as artworks. We compare our approach quantitatively and qualitatively with existing baselines and state-of-the-art techniques.
BibTeX
@inproceedings{arar2024palp,
  author = {Moab Arar and Andrey Voynov and Amir Hertz and Omri Avrahami and Shlomi Fruchter and Yael Pritch and Daniel Cohen-Or and Ariel Shamir},
  title = {{PALP: Prompt Aligned Personalization of Text-to-Image Models}},
  booktitle = {SIGGRAPH Asia},
  year = {2024},
  doi = {10.1145/3680528.3687604},
}
Figure