Mapping Visual Instances to Unique Text Tokens

This direction is merged into another direction: Janus problem, identity-preserving textual inversion. Its last version stays readable and citable here.

Version 1 of 115 papers23 conceptsfrozen 2026-10-04

When you ask a generative model to create an image of a "golden retriever," it samples from a broad, probabilistic "average" of every retriever it saw during training. This works for generic scenes, but it fails when you need a specific instance—your own dog, a particular antique chair, or a unique character. Without a way to anchor the model to a specific identity, the generative process suffers from "generic drift," where the unique textures and structures of a reference object are replaced by the model's internal prototype of that category.

To solve this, researchers have converged on a strategy: optimizing a unique, instance-specific text token that captures the fine-grained visual identity of a single reference image. By mapping a specific object to a modular textual identifier, we can treat complex visual identities as words that can be recomposed, edited, or projected into 3D.

The Problem of Generic Drift

In tasks like 3D reconstruction or image editing, the model must "hallucinate" new information—such as the back of an object or a new pose—while keeping the original identity intact. Standard diffusion models struggle here because their text encoders are trained on general concepts.

As Luke Melas-Kyriazi and colleagues observe in RealFusion, using a generic label like "a toy" for 3D reconstruction causes the model to default to a common version of that toy on its hidden sides. To prevent this, they optimize a unique token to represent the specific input object. This ensures that the generative prior remains conditioned on the specific features of the target rather than drifting toward a category average.

Learning Identity through Textual Inversion

The primary mechanism for this anchoring is a localized version of Textual Inversion. Instead of requiring a large dataset, these methods optimize a single embedding vector so that the model reconstructs the specific reference image when prompted with that token.

In the context of 3D generation, Guocheng Qian and Aliaksandr Siarohin demonstrate in Magic123 that learning an instance-specific token is critical for preventing color shifts and maintaining visual fidelity across different viewpoints. Similarly, Senthil Purushwalkam and Nikhil Naik use this technique in ConRad to ensure that hallucinated views of an object remain semantically and visually consistent with the original 2D reference.

Modular Composition and Layered Control

Beyond 3D, this "tokenization" of identity allows for sophisticated image composition. In Collage Diffusion, Vishnu Sarukkai, Christopher Re, and their team introduce a layer-based interface. By learning a unique modifier token for each layer (e.g., a specific vase or a specific flower), the system can harmonize these disparate elements into a single scene while preserving the fine-grained appearance of each individual component.

This modularity is also key to precise editing. Xudong Liu and the S2Edit team show that by optimizing a unique token for a subject through localized fine-tuning, they can change a person's expression or surroundings without losing the unique facial features that define their identity.

Consistency Across Time and Space

The ultimate goal of these methods is "identity persistence"—the ability for a subject to remain the same across different prompts, views, or frames.

  • In Storytelling: Ji‐Hun Park and H. S. Jo in Infinite-Story extract a subject's embedding from a reference prompt and substitute it into all subsequent prompts. This ensures the text encoder provides a mathematically identical representation of the character, even as the scene description changes.
  • In 3D Loops: Zhenyu Tang and colleagues in Cycle3D use reference-view injection to feed features from the original image back into the generation process, creating a "generation-reconstruction" cycle that anchors the 3D geometry to the source identity.

By moving from generic category labels to instance-specific tokens, these researchers have turned diffusion models from "imaginative" engines into precise tools for reconstruction and storytelling, where the model's vast prior knowledge is constrained by the unique reality of a single image.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mapping_visual_instances_to_unique_text__fc4a48e6,
  title = {Mapping Visual Instances to Unique Text Tokens},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mapping-visual-instances-to-unique-text-tokens/rdr_fc4a48e6803e43e3b0f5b1a8473b6b17},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mapping_visual_instances_to_unique_text__fc4a48e6_v1,
  title = {Mapping Visual Instances to Unique Text Tokens},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mapping-visual-instances-to-unique-text-tokens/rdr_fc4a48e6803e43e3b0f5b1a8473b6b17/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}