Abstract
The development of large language models (LLMs) has significantly enhanced the capabilities of multimodal LLMs (MLLMs) as general assistants. However, lack of user-specific knowledge still restricts their application in human's daily life. In this paper, we introduce the Retrieval Augmented Personalization (RAP) framework for MLLMs' personalization. Starting from a general MLLM, we turn it into a personalized assistant in three steps: (a) Remember, (b) Retrieve, and (c) Generate. Unlike previous methods, RAP allows real-time concept editing via updating the external database. To further improve generation quality and alignment with user-specific information, we design a pipeline for data collection and create a specialized dataset for personalized training of MLLMs. Our models demonstrate outstanding flexibility and generation quality across a variety of tasks, such as personalized image captioning, question answering and visual recognition.
Introduction
While general-purpose MLLMs like LLaVA or GPT-4V excel at broad visual recognition, they are effectively "amnesiacs" regarding a user's private world. They can identify a generic dog, but they cannot inherently know that a specific animal is the user's pet, "Lala," who has unique preferences. Existing personalization methods typically rely on computationally expensive fine-tuning or learning new embeddings for specific objects, which fails to scale as a user's personal context grows.
RAP (Retrieval-Augmented Personalization) addresses this by treating personalization as a retrieval problem rather than a learning problem. This allows the model to recognize and reason about user-specific concepts—friends, pets, or personal items—without requiring any fine-tuning for new concepts.
The RAP Framework
The RAP framework operates through a three-stage pipeline designed for efficiency and flexibility:
- Remember: Users provide a single reference image and a brief description (e.g., "This is my cat, Luna"). This information is stored in a key-value database where the "key" is a visual feature vector and the "value" is the image-text pair.
- Retrieve: When a query image is received, the system uses an open-world detector (YOLO-World) to identify regions of interest. These regions are matched against the database using Euclidean distance to pull the most relevant personal concepts.
- Generate: The query, retrieved images (as visual tokens), and retrieved text descriptions are fed into the MLLM. The model is trained to ground its answers in this retrieved context.
Dataset Construction and Training
To train MLLMs to effectively use retrieved personal data, the authors developed the RAP Dataset, a collection of 260K samples covering visual grounding, recognition, and QA. They repurposed existing datasets (RefCOCO, TAO, Object365) and used Gemini-1.5 for high-quality personalized annotations. To ensure robustness across different viewpoints, tools like Inpaint-Anything and Wonder3D were used to generate synthetic 3D views and diverse backgrounds for training concepts.
Experimental Results
RAP demonstrates significant advantages over existing baselines like MyVLM and Yo'LLaVA:
- Zero-Shot Personalization: RAP works with just one positive image and zero additional training, whereas previous methods required extensive negative sampling and multiple positive examples.
- Real-time Knowledge Editing: Because knowledge resides in an external database, updates are instantaneous. Changing a pet's favorite food in the database immediately reflects in the model's next response.
- Superior Performance: In personalized captioning, RAP-LLaVA achieved an F1-score of 94.97, outperforming MyVLM's 85.50. Performance remains robust even as the database scales to hundreds of concepts.
Conclusion
RAP provides a scalable, flexible solution for MLLM personalization. By decoupling personal knowledge from model weights, it enables real-time updates and zero-shot recognition of an "infinite" number of concepts. Future work will focus on optimizing context window usage and improving retriever robustness to handle fine-grained differences between similar personal items.
Summary updated