Instance Level Personalization for Vision Language Models

This direction is split into other directions: instance-level personalization. Its last version stays readable and citable here.

Version 1 of 145 papers45 conceptsfrozen 2026-10-04

Standard Vision-Language Models (VLMs) are world-class generalists. If you show a model like CLIP or LLaVA a photo of a golden retriever, it can easily identify "a dog." However, if you ask it to find "Fido"—your specific golden retriever—the model typically fails. This "instance blindness" occurs because these models are trained on massive datasets labeled with broad semantic categories rather than unique individual identities.

Recent research has shifted toward Personalization, a set of techniques designed to bridge this gap. The goal is to allow a frozen, pre-trained model to recognize and reason about specific, user-defined entities—like a favorite toy, a family member, or a personal vehicle—using only a handful of reference images.

Expanding the Vocabulary with Custom Embeddings

The most common intuition for personalization is to treat a specific object as a new "word" in the model's vocabulary. Instead of fine-tuning the billions of parameters in a large model, researchers like Niv Cohen and Yuval Atzmon in PALAVRA propose expanding the input vocabulary with custom word embeddings.

When you provide 3–5 images of your pet, the system learns a new embedding vector that represents that specific instance. This vector lives in the same space as words like "dog" or "cat," but it captures the unique visual nuances of your pet. Yuval Alaluf and colleagues extended this in MyVLM by using "concept heads" to help the model reason about these new tokens in complex queries, such as "my dog sitting on a red sofa."

By keeping the underlying model frozen and only learning these "soft prompts," the system avoids "catastrophic forgetting"—the tendency for AI to lose its general knowledge when retrained on specific data.

Retrieval as External Memory

Learning new embeddings for every object is computationally expensive if a user has hundreds of personal items. To solve this, researchers have turned to Retrieval-Augmented Personalization (RAP).

In the RAP framework, personal concepts are stored in an external key-value database. When a user asks a question, the model retrieves the relevant visual features from this "memory" on-the-fly. Haoran Hao and Jiaming Han demonstrate in Remember, Retrieve and Generate that this allows a model to handle an effectively infinite number of new concepts without any retraining. This turns the personalization problem into a search problem: the model doesn't need to "know" the object; it just needs to know how to look it up.

Visual Prompting and In-Context Learning

Another direction avoids weight updates entirely by using the model's existing "in-context learning" abilities. If you show a model a "mugshot" of a person and then a group photo, can it find that person?

Renjie Pi and Jianshu Zhang introduced Personalized Visual Instruction Tuning (PVIT), which trains models to use reference images as prefixes in a conversation. Similarly, in robotics, Sangoh Lee developed Visual Attentive Prompting (VAP), which uses visual "highlights" to tell a robot exactly which cup is "my cup" among several identical-looking distractors.

This approach treats personalization as a matter of focus. The model already has the visual features necessary to distinguish between two dogs; it simply needs a mechanism to prioritize the features of the reference image over the general "dog-ness" it usually looks for.

From Recognition to Action

Personalization is not just about labeling images; it is a prerequisite for useful AI assistants in the physical world.

  • Navigation: Luca Barsellotti and Rita Cucchiara argue in Personalized Instance-based Navigation that for embodied agents to be useful, they must navigate to a specific backpack, not just any backpack in a house.
  • Segmentation: Renrui Zhang showed with PerSAM that general segmentation tools can be "locked" onto a specific private concept with a single shot, allowing for precise editing or tracking of personal items.
  • Search: Andrew Zisserman and Bruno Korbar developed $\pi$-map, a lightweight mapping network that enables complex natural language searches like "my dog Chia with a stick" within personal photo libraries.

By moving from category-level to instance-level understanding, these researchers are transforming VLMs from generic encyclopedias into assistants that understand the unique, private context of an individual user's life.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_instance_level_personalization_for_visio_12fbb3f0,
  title = {Instance Level Personalization for Vision Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instance-level-personalization-for-vision-language-models/rdr_12fbb3f01a524c1897341a4204cbb439},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_instance_level_personalization_for_visio_12fbb3f0_v1,
  title = {Instance Level Personalization for Vision Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instance-level-personalization-for-vision-language-models/rdr_12fbb3f01a524c1897341a4204cbb439/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}