RAGAR: Retrieval Augment Personalized Image Generation Guided by Recommendation

Run Ling, Wenji Wang, Yuting Liu, Guibing Guo, Linying Jiang, Xingwei Wang

Abstract

Personalized Image Generation (PIG) aims to synthesize images that align with a user's unique visual preferences based on their interaction history. However, existing methods often suffer from historical noise—treating all past interactions as equally relevant—and a "consistency trap," where models prioritize semantic fidelity to a reference image over the user's actual style. We propose RAGAR (Retrieval Augmented Personalized Image GenerAtion guided by Recommendation), a framework that integrates a retrieval module to filter relevant historical items, a generation module using LLM-extracted keywords, and a reflection module that utilizes a recommendation-style ranking model as a reward signal. Experiments on three real-world datasets (POG, ML-latest, SER30K) demonstrate that RAGAR significantly outperforms state-of-the-art models in generating personalized images that users prefer while maintaining core semantic meaning.

Introduction

Personalized Image Generation (PIG) seeks to take a reference item (e.g., a "short-sleeved shirt") and render it in a style a specific user loves (e.g., "pink, sporty, floral"). Current approaches face two primary hurdles. First, Historical Noise: models often fail to distinguish between relevant and irrelevant past interactions. For instance, a user's past purchase of a laptop should not influence the design of a new t-shirt. Second, the Consistency Trap: most models optimize for pixel-level similarity to the reference, leading to "underfitting" the user's unique flair to ensure the object remains recognizable.

Overview of the RAGAR framework showing how retrieval improves style alignment and how recommendation ranking guides the generation process.

Methodology

RAGAR introduces a three-stage pipeline designed to bridge the gap between recommendation systems and generative models:

1. Retrieval Module

To combat historical noise, RAGAR employs CLIP and BLIP-2 to identify the "top-k" historical items most semantically similar to the reference item. This ensures that style cues are derived from relevant past interactions (e.g., using past clothing items to inform a new apparel design) rather than the user's entire, potentially diverse, history.

2. Generation Module

The framework utilizes a Large Language Model (LLM), specifically Qwen2.5, to extract and summarize stylistic keywords from the retrieved history. These keywords are fused with global preferences through a "Balance Calibrator," which feeds into a diffusion-based generator (Stable Diffusion 1.5) to produce the final image.

3. Reflection Module

Acting as a "critic," this module uses a pre-trained multi-modal ranking model (MICRO) to evaluate generated images. Instead of simple pixel matching, RAGAR uses Policy Gradients to reward the generator when it produces an image that would rank higher in a recommendation system for that specific user. This aligns the generative process directly with user preference metrics.

Experimental Results

The authors evaluated RAGAR on three diverse datasets: POG (fashion), ML-latest (movie posters), and SER30K (stickers/emojis).

Quantitative and Qualitative Performance

RAGAR achieved state-of-the-art (SOTA) results across all metrics. In the POG dataset, it significantly improved the Rank Change ($\Delta R$), a metric indicating how much the generated image improves the user's preference ranking compared to baselines. Human evaluations consistently ranked RAGAR as superior, noting its ability to maintain the reference theme (e.g., a "cat and guitar") while perfectly capturing personal aesthetic preferences (e.g., specific color schemes).

The Impact of Retrieval

A key finding was that relevance is more important than volume. Using fewer but more relevant historical items (approximately $k=5$) produced better results than using the entire history, which often introduced confusing noise.

Comparison of different retrieval strategies showing that targeted retrieval (Ret) significantly outperforms random selection and expanded retrieval in Recall@10 and NDCG@10.

Conclusion

RAGAR successfully demonstrates that treating personalization as a recommendation problem can significantly enhance generative AI. By using a ranking model as a reward signal and filtering history for relevance, the system captures latent user interests more effectively than traditional pixel-matching approaches. Future work will focus on more efficient gradient propagation through the diffusion process to allow for even finer control over the balance between prompt fidelity and personal aesthetic.

Summary updated

BibTeX
@inproceedings{ling2026ragar,
  author = {Run Ling and Wenji Wang and Yuting Liu and Guibing Guo and Linying Jiang and Xingwei Wang},
  title = {{RAGAR: Retrieval Augment Personalized Image Generation Guided by Recommendation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i18.38553},
}
Figure