Remember, Retrieve and Generate: Understanding Infinite Visual Concepts as Your Personalized Assistant

Haoran Hao, Jiaming Han, Changsheng Li, Yu-Feng Li, Xiangyu Yue

Introduction

Multimodal Large Language Models (MLLMs) like LLaVA have demonstrated remarkable general reasoning capabilities. However, they typically lack "user-specific knowledge"—the ability to recognize a user's specific pet, car, or personal items. Retrieval-Augmented Personalization (RAP) is a novel framework designed to bridge this gap, allowing MLLMs to identify and reason about an infinite number of user-defined concepts using only a single reference image per concept.

The Personalization Problem

Existing methods for personalizing MLLMs generally fall into two categories: fine-tuning and special token embeddings. Fine-tuning is computationally expensive and slow, making it impractical for real-time updates. Special token methods often require numerous "positive" and "negative" images to avoid overfitting or interference between concepts. RAP addresses these limitations by being data-efficient (requiring only one image), flexible (allowing real-time database edits), and highly scalable.

The RAP Framework

The RAP framework decouples personal memory from the model's weights by utilizing an external database. The process operates in three distinct stages:

  1. Remember: User-specific concepts are stored in a key-value database. The "key" is a visual feature vector generated by a CLIP encoder, while the "value" contains the original image and a descriptive text metadata.
  2. Retrieve: When a query image is provided, an open-world detector (e.g., YOLO-World) identifies regions of interest. These regions serve as queries to pull the most relevant concept pairs from the database using a multimodal retriever (FAISS).
  3. Generate: The MLLM receives the user's query alongside the retrieved context. The model is specifically instruction-tuned to integrate this external knowledge into its final response.
The RAP-MLLM framework illustrates the three-stage process: Remember (storing concepts in a user database), Retrieve (fetching relevant data), and Generate (producing personalized responses).

Data and Training

To enable effective context integration, the researchers developed a massive Personalization Dataset of 260,000 samples. This dataset covers visual grounding, recognition, captioning, and question answering. High-quality annotations were generated using Gemini-1.5. To ensure robustness, the team employed data augmentation (like 3D modeling via Wonder3D) to simulate novel views from a single image and included "noise concepts" during training to teach the model to ignore irrelevant retrieved data.

Key Results and Scalability

Evaluations of RAP-LLaVA and RAP-Phi3-V show significant performance gains over existing baselines. In personalized captioning tasks, RAP-LLaVA achieved an F1-score of 94.97, outperforming MyVLM (85.50).

Crucially, the system demonstrates "Infinite Scalability." Even as the user database grows to 500 concepts, the retriever maintains a Top-5 recall above 90%. Because the model relies on retrieved context rather than internalized weights, it can handle a virtually unlimited number of new visual identities without performance degradation or the need for retraining.

Conclusion and Future Work

RAP represents a significant step toward truly personalized AI assistants. By moving personal knowledge to an external database, it achieves real-time editing and superior data efficiency. While the system is currently limited by the computational overhead of processing retrieved tokens and the accuracy of the initial retriever, future work aims to optimize context window usage and improve multimodal retrieval precision for highly similar objects.

Summary updated

BibTeX
@inproceedings{hao2025remember,
  author = {Haoran Hao and Jiaming Han and Changsheng Li and Yu-Feng Li and Xiangyu Yue},
  title = {{Remember, Retrieve and Generate: Understanding Infinite Visual Concepts as Your Personalized Assistant}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=fo0VxfNe8U},
}
Figure