Abstract
Personalizing text-to-image (T2I) models to represent specific subjects often results in a significant trade-off: the model either captures the subject's identity with high fidelity but ignores the prompt's context, or follows the prompt while losing the subject's unique characteristics. We present PALP (Prompt Aligned Personalization), a framework that utilizes a novel score distillation technique to maintain alignment with complex, multi-element instructions during the fine-tuning process. By regularizing the personalization process with a Delta Denoising Score (DDS), PALP enables the generation of images that are both faithful to a specific subject and strictly adherent to intricate textual prompts.
Introduction
Large-scale text-to-image models, such as Stable Diffusion, have demonstrated a remarkable ability to generate diverse imagery from natural language. However, these models lack the inherent knowledge of specific, user-provided subjects (e.g., a specific pet, a personal item, or a unique building). Existing personalization techniques like DreamBooth and Textual Inversion attempt to bridge this gap by fine-tuning models on a small set of reference images.
Despite their success, these methods frequently suffer from "identity vs. prompt" interference. When a model is heavily optimized to replicate a specific subject, it often overfits, causing it to "forget" its prior knowledge of styles, lighting, and spatial relationships. For instance, a model trained on a specific cat might generate that cat perfectly but fail to place it in a "charcoal sketch style on the moon" because the subject's features overwrite the stylistic and contextual cues of the prompt. PALP addresses this limitation, allowing for high-fidelity personalization within highly specific and complex scenes.
Methodology
PALP operates by optimizing the model through two parallel paths during the training phase, ensuring that subject learning does not come at the cost of prompt adherence.
The Personalization Path
This path follows standard personalization protocols. It utilizes Low-Rank Adaptation (LoRA) to update a targeted subset of the model's weights while simultaneously optimizing a new word embedding (a placeholder token [V]) to represent the subject. A standard reconstruction loss is applied to ensure the model learns the visual features of the reference subject.
The Prompt-Alignment Path
The core innovation of PALP is the integration of Score Distillation Sampling (SDS) as a regularizer. Specifically, the authors employ a variant known as Delta Denoising Score (DDS). During training, the framework evaluates a "clean" version of the target prompt (e.g., "A sketch of a cat" instead of "A sketch of [my cat]").
By calculating the difference (the "delta") between how the original pre-trained model denoises the clean prompt and how the personalized model denoises the subject-specific prompt, PALP forces the model to maintain the original model's understanding of style and context. This tethering mechanism prevents the subject's identity from dominating the latent space, ensuring that elements like "Paris," "Pop Art," or "Underwater" are preserved in the final output.
Experimental Evaluation
The researchers benchmarked PALP across three primary scenarios:
- Multi-shot Personalization: Utilizing standard datasets featuring animals, toys, and architecture to test general subject fidelity.
- Single-shot Personalization: Evaluating the model's ability to learn a subject (specifically human portraits) from only one reference image.
- Complex Prompt Benchmark: A specialized test set consisting of ten intricate prompts with at least four distinct elements (e.g., specific styles, times of day, and color palettes) to measure the limits of prompt alignment.
Results and Discussion
PALP demonstrates significant improvements over baseline methods such as P+, NeTI, and DreamBooth in both qualitative and quantitative assessments.
- Text Alignment: In human evaluation studies, PALP achieved a 91.2% text alignment score, significantly higher than the 73.3% achieved by the next best method (TI+DB). This indicates that PALP is uniquely capable of incorporating multiple complex prompt elements without omission.
- Subject Fidelity: The model maintained a 72.1% personalization score, confirming that the increased prompt alignment does not result in a loss of the subject's identity.
- Qualitative Performance: Visual analysis shows that PALP successfully renders subjects in challenging artistic styles, such as "Vitruvian Man" sketches or "Risograph" prints, where competing models typically fail to either maintain the style or preserve the subject's likeness.
Conclusion and Future Work
PALP effectively mitigates the trade-off between subject fidelity and prompt alignment by leveraging the prior knowledge of pre-trained models through score distillation. While the current implementation is optimized for a specific target prompt—making it a "per-scene" optimization tool—it sets a new standard for high-fidelity, complex AI art generation. Future research will focus on developing prompt-aligned adapters to enable "instant" personalization, removing the need for per-scene fine-tuning while maintaining the high alignment standards established by PALP.
Summary updated