Abstract
Personalized Image Generation (PIG) aims to synthesize images that align with a user's unique visual preferences based on their interaction history. However, existing methods often suffer from historical noise—treating all past interactions as equally relevant—and a "consistency trap," where models prioritize semantic fidelity to a reference image over the user's actual style. We propose RAGAR (Retrieval Augmented Personalized Image GenerAtion guided by Recommendation), a framework that integrates a retrieval module to filter relevant historical items, a generation module using LLM-extracted keywords, and a reflection module that utilizes a recommendation-style ranking model as a reward signal. Experiments on three real-world datasets (POG, ML-latest, SER30K) demonstrate that RAGAR significantly outperforms state-of-the-art models in generating personalized images that users prefer while maintaining core semantic meaning.
Introduction
Personalized Image Generation (PIG) seeks to take a reference item (e.g., a "short-sleeved shirt") and render it in a style a specific user loves (e.g., "pink, sporty, floral"). Current approaches face two primary hurdles. First, Historical Noise: models often fail to distinguish between relevant and irrelevant past interactions. For instance, a user's past purchase of a laptop should not influence the design of a new t-shirt. Second, the Consistency Trap: most models optimize for pixel-level similarity to the reference, leading to "underfitting" the user's unique flair to ensure the object remains recognizable.
Methodology
RAGAR introduces a three-stage pipeline designed to bridge the gap between recommendation systems and generative models:
1. Retrieval Module
To combat historical noise, RAGAR employs CLIP and BLIP-2 to identify the "top-k" historical items most semantically similar to the reference item. This ensures that style cues are derived from relevant past interactions (e.g., using past clothing items to inform a new apparel design) rather than the user's entire, potentially diverse, history.
2. Generation Module
The framework utilizes a Large Language Model (LLM), specifically Qwen2.5, to extract and summarize stylistic keywords from the retrieved history. These keywords are fused with global preferences through a "Balance Calibrator," which feeds into a diffusion-based generator (Stable Diffusion 1.5) to produce the final image.
3. Reflection Module
Acting as a "critic," this module uses a pre-trained multi-modal ranking model (MICRO) to evaluate generated images. Instead of simple pixel matching, RAGAR uses Policy Gradients to reward the generator when it produces an image that would rank higher in a recommendation system for that specific user. This aligns the generative process directly with user preference metrics.
Experimental Results
The authors evaluated RAGAR on three diverse datasets: POG (fashion), ML-latest (movie posters), and SER30K (stickers/emojis).
Quantitative and Qualitative Performance
RAGAR achieved state-of-the-art (SOTA) results across all metrics. In the POG dataset, it significantly improved the Rank Change ($\Delta R$), a metric indicating how much the generated image improves the user's preference ranking compared to baselines. Human evaluations consistently ranked RAGAR as superior, noting its ability to maintain the reference theme (e.g., a "cat and guitar") while perfectly capturing personal aesthetic preferences (e.g., specific color schemes).
The Impact of Retrieval
A key finding was that relevance is more important than volume. Using fewer but more relevant historical items (approximately $k=5$) produced better results than using the entire history, which often introduced confusing noise.
Conclusion
RAGAR successfully demonstrates that treating personalization as a recommendation problem can significantly enhance generative AI. By using a ranking model as a reward signal and filtering history for relevance, the system captures latent user interests more effectively than traditional pixel-matching approaches. Future work will focus on more efficient gradient propagation through the diffusion process to allow for even finer control over the balance between prompt fidelity and personal aesthetic.
Summary updated