TAMEing the Open-World Personalization: Towards Open-Set Personalized MLLM Assistant

Rongpei Hong, Jian Lang, Ting Zhong, Fan ZhouarXiv 2026 · arXiv

TL;DR

TAME-O is the first framework that enables Multimodal Large Language Models (MLLMs) to autonomously discover and internalize new user-specific concepts—like your favorite mug or a new pet—during natural conversation. By moving from a "closed-set" to an "open-set" paradigm, it allows AI assistants to grow their personal knowledge base without manual user registration, using a novel training-free reconciliation skill.

What Problem Does This Paper Address?

Personalized AI assistants are only useful if they actually know your stuff. Most current research into Multimodal Large Language Model (MLLM) personalization operates under a "closed-set" assumption: the model is given a fixed list of concepts (e.g., "Bob's dog," "Alice's car") in advance. In the real world, this is a dealbreaker. You don't want to manually register every new object you mention to your AI; you want it to "see" a new item, remember it, and recognize it later.

The authors identify two major hurdles in this "open-set" transition:

  1. Concept Discovery: How does the model decide if an object is something it has seen before or a brand-new entity?
  2. Concept Fission: If the model fails to recognize an existing concept (perhaps because it looks different in a new light), it might erroneously create a duplicate entry in its memory. This "fission" leads to fragmented knowledge and "memory chaos."
Figure 1: Paradigm comparison between (a) Closed-Set Personalization, where the MLLM passively acquires novel concepts from user provision, and (b) Open-Set Personalization, where the MLLM actively grasps new concepts from everyday interactions.

Methods

The researchers propose TAME-O, a plug-and-play framework that can be integrated into existing personalized MLLMs. Its core is the Concept Reconciliation Skill (CR Skill), which works through three main components:

  • Time-Aware Dynamic (TAD) Memory: Unlike static memories, TAD Memory stores concept attributes with temporal markers. This recognizes that the appearance or state of a personal object might shift over time.
  • Temporally Weighted Retrieval: When a new object appears, TAME-O searches its memory for candidates. Crucially, it uses a decay function to prioritize recent memories, which are often more reliable cues for identification than distant history.
  • MLLM-based Reconciliation: Instead of relying on a simple similarity threshold (which often fails for visually similar objects), TAME-O feeds the top candidates and the current context back into the MLLM. The model then reasons whether the object is an "existing" concept or a "novel" one.
  • Periodic Concept Fusion (PCF): To combat concept fission, the system periodically scans its memory for highly similar entries and uses the MLLM to decide if they should be merged into a single unified concept.
Figure 2: Paradigm of TAME-O. When receiving a new dialogue, the CR skill first retrieves candidate personalized concepts from the TAD Memory in a temporally weighted manner. The retrieved candidates are then provided to the MLLM, together with the user query and recent dialogue context, to determine whether the referenced concept is existing or novel, achieving open-set personalization.

Datasets

The authors evaluate TAME-O on two benchmarks:

  • LCMP-H: An existing benchmark for long-context personalization with 30 concepts.
  • LCMP-O (New): A more rigorous "Open-Set" benchmark introduced in this paper. It features 45 concepts specifically designed with high intra-class similarity (e.g., different cats of the same breed, people of similar ages) to push the limits of the model's reconciliation abilities.

Key Insights and Ideas

  • Recency is Key for Identity: The "aha moment" here is that user-specific concepts are dynamic. By weighting recent memories more heavily, the system handles attribute shifts (like a dog getting muddy or a person changing clothes) much better than a flat memory search.
  • Reasoning Over Retrieval: Simple vector similarity is too blunt a tool for fine-grained identity. Using the MLLM's own reasoning capabilities to perform the final "reconciliation" step allows the system to use linguistic context (e.g., "I bought a new hat") to override ambiguous visual signals.
  • Memory "Garbage Collection": Concept fission is inevitable in open-set settings. The PCF strategy acts as a necessary maintenance layer, ensuring the assistant's "brain" doesn't get cluttered with duplicate identities for the same object.
Figure 3: A concept fission example for concept Cat 1.

Key Results

TAME-O demonstrates a significant leap over baselines that try to adapt closed-set methods to open-world scenarios:

  • Reconciliation Accuracy: On the challenging LCMP-O dataset, TAME-O achieved a Concept Reconciliation Accuracy (CRA) of 71.49%, whereas a modified version of the previous state-of-the-art (TAME†) struggled at around 31.63%.
  • Fission Control: The PCF strategy successfully reduced the Concept Fission Rate (CFR) over time, dropping it from 22.0% to 13.8% on LCMP-O as the interaction progressed.
  • Scalability: Performance remained stable even as the number of concepts in the environment increased, a critical requirement for a long-term assistant.

Limitations and Future Work

  • Reliance on Upstream Modules: TAME-O depends on off-the-shelf grounding (like Grounding-DINO) and embedding models. Errors in these initial steps can propagate through the system.
  • Noisy Environments: The benchmarks are controlled; real-world user photos can be blurry, poorly lit, or contain ambiguous references that might confuse the reconciliation logic.
  • Privacy: Maintaining a detailed, long-term multimodal memory of a user's life raises significant privacy concerns. The authors suggest that future implementations should focus on local storage and user-controlled data deletion.

For more details, you can find the code at https://github.com/ronpay/TAME-O.

Summary updated

Abstract

Personalizing Multimodal Large Language Models (MLLMs) as intelligent assistants has recently emerged as an active research topic, where an MLLM is expected to go beyond generic, one-size-fits-all replies, and instead generate responses grounded in user-specific objects, entities, and concepts encountered during interactions. Nevertheless, existing studies mainly focus on personalizing MLLMs on a static, predefined set of personalized concepts and require users to manually and frequently extend this set to accommodate newly emerging concepts, which is infeasible in dynamic-world scenarios. To address this limitation, we recast conventional MLLM personalization as an Open-Set problem, and propose TAME-O, the first long-context open-set personalized MLLM assistant. Specifically, TAME-O introduces a training-free, plug-and-play Concept Reconciliation Skill (CR Skill). It enables the assistant to automatically unlock novel concepts during prolonged human-MLLM interactions while preventing confusion with previously unlocked ones, allowing the MLLM to continually grasp more and more concepts and progressively refine the modeling of known ones. Extensive experiments under the open-set personalization setting underscore the efficacy of our method.
BibTeX
@misc{hong2026tameing,
  author = {Rongpei Hong and Jian Lang and Ting Zhong and Fan Zhou},
  title = {{TAMEing the Open-World Personalization: Towards Open-Set Personalized MLLM Assistant}},
  year = {2026},
  eprint = {2611.14118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14118},
}
Figure