TL;DR
Online repositories host tens of thousands of personalized text-to-image models, yet users have no standardized way to understand what changed under the hood. FineXL introduces a training-free framework that extracts the distributional shift between a base model and its fine-tuned counterpart, decomposing this shift into a set of orthogonal natural language concepts paired with quantitative intensity scores. Across diffusion, GAN, and autoregressive models, FineXL outperforms existing vision-language baselines by over 50% in explaining both the nature and degree of personalization.
What Problem Does This Paper Address?
Personalizing foundation image generators (via methods like DreamBooth or LoRA) is ubiquitous, resulting in over 80,000 community models on HuggingFace alone. However, documentation for these models is often sparse or missing. When a user wants to select a model, they must resort to tedious, ad-hoc trial and error.
Existing explainability approaches fall short:
- Mechanistic interpretability (e.g., sparse autoencoders) is computationally prohibitive and tied to specific architectures.
- Visual feature extraction highlights differences visually (e.g., patch clusters), but humans cannot easily interpret what concept those patches unify.
- Vision-Language Model (VLM) image difference captioning (e.g., VisDiff, GSCLIP) produces vague, qualitative summaries (e.g., "a modern artistic style") that cannot disentangle multiple simultaneous stylistic shifts or quantify the level (strength) of personalization.
Methods
FineXL operates without retraining or fine-tuning, leveraging the linear representation hypothesis in multimodal embedding spaces:
[Base Model] --> Enc_img ─┐
├─> Distribution Divergence Vector (V_div)
[Personalized Model] --> Enc_img ─┘ │
▼
[Pairwise Outputs] --> VLM ──> Candidate Concepts {C_i}
│
▼
[Aligned Text Enc.] ──> Orthogonality Filter ──> Linear Decomposition:
(removes redundancy) V_div ≈ ∑ w_i · f(C_i)
- Quantifying Distributional Divergence ($V_{div}$): Generates image pairs from base ($G_{base}$) and fine-tuned ($G_{personal}$) models using sampled text prompts, mapping their expected difference into vector space via an aligned image encoder (e.g., CLIP).
- Low-Level Concept Discovery: Prompts a VLM (e.g., GPT-4o) with pairs of base/personalized images to summarize distinct visual shifts into lists of concise adjectives or short phrases.
- Concept Vectorization: Converts each discovered text concept $C_i$ into the shared representation space by measuring prompt differences: $f(C_i) = \sum \{Enc_{text}(t \cup C_i) - Enc_{text}(t)\}$.
- Orthogonality Filtering: Prunes redundant concepts (e.g., "vivid" vs. "vibrant") by projecting candidate concept vectors onto existing ones, retaining only orthogonal, semantically distinct features.
- Linear Decomposition: Decomposes $V_{div}$ into a linear combination of retained concept vectors ($\sum w_i f(C_i)$) until the reconstruction residual drops below a stopping threshold ($\epsilon_{decomp} = 0.2$). The coefficients $w_i$ serve as fine-grained, quantitative scores indicating the strength of each personalization aspect.
Datasets
- Synthetic Style Dataset: 400 images across 15 distinct styles generated via Stable Diffusion 3.5 using MS COCO prompts concatenated with known style descriptions to provide clear ground truth.
- Image Style Transfer Dataset: 2,500 images across 50 unique artistic styles with BLIP-2 captions.
- WikiArt Dataset: 81,444 fine-art paintings across 129 artists; the top 15 most represented artists were chosen as distinct target personalization styles.
- Latent Manipulation (Facial Features): Facial attribute editing (e.g., adding a mustache) with controlled editing scale parameters used as ground truth for single-attribute continuous scaling.
Key Insights and Ideas
- Linear Additivity of Style Shifts: Distributional differences between generative models reside along approximately linear subspaces in aligned multimodal embeddings (like CLIP), confirming that complex visual shifts can be modeled as additive concept vectors.
- Orthogonality Eliminates Synonym Redundancy: VLMs naturally suggest overlapping descriptors. Enforcing vector orthogonality automatically prevents collinearity and isolates distinct axes of variation (e.g., separating "brushstroke texture" from "color saturation").
- Training Progression Directly Maps to Coefficients: In single-aspect experiments, FineXL’s derived weights grew monotonically alongside the number of fine-tuning steps, effectively reading out training duration and stylistic convergence without accessing model checkpoints or training logs.
Key Results
- Single-Aspect Intensity Tracking: When estimating varying personalization levels (1 to 10 training steps), FineXL achieved a Mean Absolute Error (MAE) of 0.7 on the synthetic set, 1.6 on Style Transfer, and 1.4 on WikiArt—slashing the error of competitive baselines like VisDiff (1.6 / 2.5 / 2.2) and GSCLIP (1.8 / 2.6 / 2.5) by 56%.
- Multi-Aspect Mixture Disentanglement: In models trained on data mixtures of two distinct styles at various ratios, FineXL correctly identified the mixture coordinates with 94.4% accuracy on both Synthetic and Style Transfer datasets, compared to 77.8% and 88.9% for VisDiff.
- Model Agnostic Generalizability: FineXL transferred seamlessly beyond diffusion models:
- ControlGAN: FineXL reached an MAE of 0.9 (Synthetic) and 1.4 (WikiArt), outperforming VisDiff (1.6 / 2.2).
- Anole-7B (Autoregressive): FineXL achieved an MAE of 1.5 (Synthetic) and 0.7 (Style Transfer), consistently leading over baselines.
Limitations and Future Work
- Non-Linear Embedding Discrepancies: While text-image encoders provide strong linear approximations, certain complex transformations (e.g., latent-space facial geometry manipulations) are not strictly linear in CLIP space, leading to occasional scaling distortions.
- Residual "Dark Matter": Non-decomposable components of the distribution shift cannot be expressed by discrete language tokens; setting the decomposition residual threshold too low ($\le 0.05$) causes the algorithm to stall or extract noisy, non-meaningful concepts.
- Encoder Alignment Bottlenecks: Performance remains bounded by the representation fidelity of existing dual encoders. While SD 3.5 and standard CLIP performed best, domain gaps outside COCO-style distributions could affect concept vector alignment.
Summary updated