GDP: Graph-Based Dynamic Personalization for Multimodal Large Language Models

Cong Ray, Xiangwen Deng, Feice Huang, ZhengXian Wu, Shen'ao Jiang, Peng Jiao, Zhifang Liu, Haoqian WangCVPR 2026 Findings 2026 · OpenReview

TL;DR

The paper introduces Graph-based Dynamic Personalization (GDP), a framework that allows Multimodal Large Language Models (MLLMs) to adapt to evolving user preferences in real time. By storing user interactions in a dynamically weighted memory graph, GDP overcomes the limitations of static user profiles and fragile face-recognition systems, achieving state-of-the-art performance in personalized question answering and identity recognition.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) excel at general vision-language tasks, making them truly personalized assistants remains a challenge. Current personalization approaches suffer from two major flaws:

  1. Fragile User Identification: Most systems rely strictly on face recognition to identify the user. If the user's face is occluded, turned away, or poorly lit, the system fails.
  2. Static User Modeling: Existing methods treat user preferences as fixed, predefined profiles. In reality, human preferences are dynamic and evolve over time through interactions. Standard Retrieval-Augmented Generation (RAG) systems rank information purely by semantic similarity, failing to capture the strength or recency of a user's changing tastes.

As illustrated in Figure 1, the authors tackle these issues by shifting personalization research from static identity recognition to evolving preference modeling.

Figure 1. GDP is the first multimodal benchmark for dynamic user preference modeling, shifting personalization research from static identity recognition to evolving preferences.

Methods

The GDP framework consists of three core components that manage the lifecycle of user preferences:

  • Personalized Extraction: A supervised fine-tuned Personalization Enhancement (PE) Agent (built on LLaVA) extracts preference cues from multimodal inputs and estimates a "preference confidence" score. Crucially, if face recognition fails due to poor visual conditions, the agent uses a Graph-Enhanced Entity Recognition strategy, inferring identity from historical graph nodes like previously worn clothing, accessories, or frequent locations.
  • Graph Update & Personalization Storage: Extracted information is stored in a multi-user dynamic graph memory containing User Nodes, Likeness Nodes, and contextual nodes (places, objects, activities). Edges between nodes are updated dynamically using a custom weight mechanism ($w_{ij}$) that factors in extraction confidence, semantic distance, co-occurrence frequency, and temporal decay ($\Delta t$) to phase out stale preferences.
  • Personalized Retrieval: When a user asks a question, a Graph Rerank module scores candidate nodes by balancing semantic similarity, node confidence, and structural relevance (the average edge weight connecting a node to its neighbors). This ensures the generated response aligns with both immediate intent and long-term preferences.

The complete system architecture is detailed in Figure 2.

Figure 2. Overview of our framework. Our method has three important modules: Personalized Extraction, Graph Update, and Personal- ized Retrieval. Through the complete pipeline, our system realizes dynamic storage of personalized information and supports high-quality personalized response generation based on user preferences.

Datasets

Because no benchmark existed for dynamic multimodal personalization, the authors built the GDP Dataset using a specialized data collection pipeline (shown in Figure 3).

Figure 3. The Dynamic Data Collection Pipeline generates three datasets: Personal Data, QA Dataset, and SFT Dataset. Raw multimodal inputs are first processed into structured Personal Data, which then forms the basis for constructing the PQA and SFT datasets.

The pipeline processes raw video and subtitle data from TV dramas and online videos to generate three distinct subsets:

  • Personal Data: Aligned multimodal interaction records sorted sequentially by time to simulate realistic dialogue scenarios.
  • SFT Dataset: Data designed to train the PE Agent to output structured graph nodes from multimodal dialogue histories.
  • PQA (Preference Question Answering) Dataset: A evaluation set containing 835 question-answer pairs. It is split into Consistency PQA (checking if the model retains existing info) and Update PQA (checking if the model accurately absorbs newly updated preferences).

Key Insights and Ideas

  • Contextual Identity Rescue: Relying solely on a face recognition model is a bottleneck. By linking visual concepts to a user node in a graph, peripheral cues (like a unique jacket or a favorite coffee shop) can successfully verify identity when the face is hidden.
  • Semantic Similarity is Not Enough: Standard RAG often retrieves text that matches the meaning of a query but ignores whether the user actually cares about it. Dynamic edge weights act as a proxy for preference intensity.
  • The Hyperparameter Balance: Ablation studies revealed that the key parameters controlling semantic noise ($\alpha$), temporal decay ($\lambda$), and frequency ($\beta$) follow an inverted U-shape curve (Figure 5). Peak performance requires a delicate balance: decaying old information without completely erasing long-term traits, and boosting frequent behaviors without amplifying noise.
Figure 5. Ablation Study of key hyperparameters α, λ, and β in Graph Update Mechanism, evaluated on the PQA Dataset.

Key Results

The authors evaluated GDP against strong baselines, including LLaVA, Yo'LLaVA, and RAP, across two main tasks:

1. Visual Recognition

GDP achieved a Weighted Accuracy of 0.956, significantly outperforming the next best training-free method, RAP + Prompt (0.809), and the fine-tuned Yo'LLaVA (0.833). It demonstrated high accuracy on both positive cases (target present) and negative cases (target absent), proving it doesn't just rely on learned answer patterns.

2. Preference Question Answering (PQA)

GDP achieved a Weighted Score of 0.834 on the PQA task, outperforming the second-best method (RAP + Prompt at 0.624) by 21%. It maintained high scores across both Consistency (0.832) and Update (0.835) metrics, proving its unique capability to handle preference shifts.

Limitations and Future Work

The primary limitation of GDP is its heavy reliance on continuous, personalized visual and interaction data, which inherently raises user privacy concerns. The authors note that storing detailed user histories (locations, clothing, preferences) in a graph structure requires robust security.

To address this, they suggest that future work should explore integrating privacy-preserving machine learning mechanisms, such as differential privacy or federated learning, to safeguard sensitive user data while maintaining dynamic personalization capabilities.

Summary updated

Abstract

Multimodal large language models (MLLMs) have demonstrated strong cross-modal capabilities, and recent work has begun incorporating user identity and preference modeling to enhance personalization. However, existing approaches struggle to reliably recognize users in complex visual conditions (e.g., occlusion, non-frontal views) and rely on static user profiles without dynamically updating preferences, limiting their adaptability to real-world evolving user states. To address these challenges, we propose Graph-based Dynamic Personalization (GDP), a framework that applies preference confidence as dynamic weights to model user preferences in real time. Specifically, the system employs a Personalization Enhancement (PE) Agent to extract preference cues, estimate preference confidence, and leverage graph associations to enhance identity recognition under challenging conditions. Second, the dynamic user preference representations are stored and updated through a weighted graph. Finally, the system retrieves relevant preference-aware content to generate responses aligned with both immediate intent and long-term preferences. In addition, we construct the GDP dataset, the first benchmark dedicated to dynamic multimodal personalization, along with a complete data construction pipeline. Experimental results show that GDP achieves state-of-the-art performance on personalized multimodal tasks such as identity recognition and personalized question answering.
BibTeX
@inproceedings{ray2026gdp,
  author = {Cong Ray and Xiangwen Deng and Feice Huang and ZhengXian Wu and Shen'ao Jiang and Peng Jiao and Zhifang Liu and Haoqian Wang},
  title = {{GDP: Graph-Based Dynamic Personalization for Multimodal Large Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=nxJrUEkZdo},
}
Figure