Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Models

Haoming Wang, Wei Gao

Abstract

The proliferation of personalized image generation models has created a "model zoo" where users struggle to understand the specific stylistic nuances of thousands of fine-tuned variants. We present FineXL, a training-free framework that provides fine-grained, natural language explanations for model personalization. By decomposing complex style shifts into quantified, orthogonal concepts, FineXL allows users to see exactly which aspects of a model were changed (e.g., "vividness" vs. "abstraction") and by how much. Our results show that FineXL significantly outperforms existing state-of-the-art captioning and explanation methods in both accuracy and multi-aspect precision.

The Challenge of the Model Zoo

The explosion of open-source AI has led to a significant discovery problem. On platforms like HuggingFace, there are over 80,000 personalized versions of models like Stable Diffusion, each fine-tuned to capture a specific artist's style, a certain lighting "vibe," or a particular character.

The "model zoo" problem: Users are often overwhelmed by the sheer number of personalized model variants without clear descriptions of their differences.

However, these models are often "black boxes." A creator might label their model as "Artistic Style," but what does that actually mean? Is it more colorful? Are the lines more blurred? Currently, users have to manually test dozens of models to find the right one—a process that is both time-consuming and subjective. Existing "explainable AI" (XAI) tools are either too technical for average users or too vague. There is a distinct lack of tools that can provide quantitative, natural language descriptions like: "This model is 80% more vivid and 30% more abstract than the original."

Methodology: Quantifying the "Vibe"

FineXL treats personalization as a mathematical decomposition problem. The workflow follows four main steps to translate visual shifts into human-readable scores.

Personalization involves fine-tuning a base model on different data mixtures to create distinct versions (V1, V2, V3). FineXL aims to explain these resulting differences.

1. Quantifying the Shift

First, the system generates images from both the "base" model and the "personalized" model using the same prompts. It uses an image encoder (like CLIP) to calculate the "distributional divergence"—essentially a vector representing the total change in visual style between the two models.

2. Concept Discovery

A Vision-Language Model (VLM), such as GPT-4o, looks at pairs of images and suggests natural language descriptors (e.g., "vibrant," "minimalist," "sketch-like") that explain the differences. This step ensures the explanations are grounded in human language.

3. The Linear Representation Hypothesis

This is the core innovation. The authors leverage the idea that complex styles can be represented as a linear combination of simpler concepts. They convert the VLM’s text descriptors into vectors in the same embedding space as the images, allowing for direct comparison between text and visual shifts.

4. Orthogonal Decomposition

To ensure the explanation isn't redundant (e.g., using both "vivid" and "bright"), FineXL selects "orthogonal" concepts—descriptors that represent distinct, non-overlapping directions in the style space. It then calculates "scores" (coefficients) for each concept to show how much each one contributes to the overall personalization.

Experimental Evaluation

The researchers evaluated FineXL using three distinct types of data to ensure robustness across different domains:

  • Synthetic Dataset: 400 images across 15 unique, manually crafted styles (e.g., "Cyberpunk," "Oil Painting") used to establish a ground truth for controlled testing.
  • Style Transfer Dataset: 2,500 images across 50 unique styles to test how well the system handles diverse artistic transformations.
  • WikiArt Dataset: A massive collection of 81,444 paintings from 129 different artists, providing a rigorous real-world test for identifying subtle artistic nuances.

Key Results and Insights

FineXL significantly outperformed existing state-of-the-art captioning and explanation methods (like VisDiff and GSCLIP):

  • Accuracy Boost: In scenarios where a model was personalized in a single aspect, FineXL improved the accuracy of the explanation by 56% compared to baselines.
  • Multi-Aspect Precision: In complex cases where a model changed in multiple ways simultaneously (e.g., becoming both more "vivid" and more "cartoonish"), FineXL reduced explanation error by at least 50%.
  • Generality: The system worked effectively across different model architectures, including Diffusion models (Stable Diffusion), GANs (ControlGAN), and Auto-regressive models (Anole-7B).

Beyond artistic style, the framework proved adaptable. By changing the VLM prompt, the authors successfully applied FineXL to "subject-driven" personalization, such as quantifying exactly how much a facial editing model adds a "mustache" or changes a "smile."

Conclusion and Future Work

FineXL demonstrates that "style" can be treated as a vector space, proving that complex artistic shifts can be broken down into additive "building blocks." As a training-free, plug-and-play method, it is immediately applicable to any new model architecture.

While the linear hypothesis works well for most styles, future work will address the slight inaccuracies caused by non-linearities in latent spaces. Additionally, the authors suggest that these fine-grained scores could eventually be used to "steer" models in real-time, allowing users to dial specific style components up or down based on the natural language explanations provided by FineXL.

Summary updated

Abstract

Image generation models are usually personalized in practical uses in order to better meet the individual users' heterogeneous needs, but most personalized models lack explainability about how they are being personalized. Such explainability can be provided via visual features in generated images, but is difficult for human users to understand. Explainability in natural language is a better choice, but the existing approaches to explainability in natural language are limited to be coarse-grained. They are unable to precisely identify the multiple aspects of personalization, as well as the varying levels of personalization in each aspect. To address such limitation, in this paper we present a new technique, namely FineXL, towards Fine-grained eXplainability in natural Language for personalized image generation models. FineXL can provide natural language descriptions about each distinct aspect of personalization, along with quantitative scores indicating the level of each aspect of personalization. Experiment results show that FineXL can improve the accuracy of explainability by 56\%, when different personalization scenarios are applied to multiple types of image generation models.
BibTeX
@misc{wang2025deciphering,
  author = {Haoming Wang and Wei Gao},
  title = {{Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Models}},
  year = {2025},
  eprint = {2511.01932},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.01932},
}
Figure