Multi-Token Textual Inversion for Fine-Grained Visual Composition

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 113 papers24 conceptsfrozen 2026-10-04

When we describe an image, we rarely use a single word. We use nouns for subjects, adjectives for textures, and verbs for states. Yet, early methods for bridging vision and language—such as Textual Inversion—tried to compress an entire visual concept into a single "pseudo-word" token.

Recent research has identified this single-token approach as a significant information bottleneck. Whether the goal is retrieving a specific product, personalizing a diffusion model, or restoring a low-resolution photo, a single vector often fails to capture the nuanced interplay between a subject and its attributes. A new wave of research proposes a more expressive alternative: decomposing images into multi-token representations.

The Information Bottleneck of Single Tokens

The core challenge in tasks like Composed Image Retrieval (CIR) is modification. If a user provides an image of a dress and asks for "3/4 sleeves and more coverage," the model must isolate the "dress-ness" from its current "sleeve-ness."

As Junda Wu and colleagues argue in Few-Shot Composition Learning for Image Retrieval with Prompt Tuning, condensing a reference image into a single visual prompt prevents the model from addressing different attributes like geometry or texture separately. Similarly, Subhadeep Koley notes in You'll Never Walk Alone that single pseudo-words limit a model's ability to encode high-dimensional structural details, such as those found in hand-drawn sketches.

By moving to multiple tokens, researchers allow the model's attention mechanism to "distribute" visual information across a sequence, much like how a sentence distributes meaning across words.

Decomposing Subject and Attribute

One of the most effective ways to use multiple tokens is through explicit semantic decomposition. In Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval, Haokun Wen and his team introduced FTI4CIR. This framework maps a reference image into a structured set of tokens: one for the primary subject and several others for specific attributes. This structural alignment mirrors human language, allowing a retrieval system to swap out an "attribute token" (like short sleeves) for a new text-guided one while keeping the "subject token" (the dress) intact.

This disentanglement is equally vital for image generation. In the Lego framework, Saman Motamed and collaborators demonstrate that complex visual states—like an object being "frozen in ice"—are too semantically dense for a single vector. By using multiple optimized word embeddings, they can capture intricate details like transparency and structural cracks without letting the identity of the original object "leak" into the style.

Granularity through Spatial and Semantic Queries

Beyond simple decomposition, multiple tokens allow models to attend to different spatial regions or levels of abstraction. Yongchao Du and the authors of Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval found that representing a query image as a sequence of learnable tokens allows different parts of the representation to focus on distinct objects. This granularity is essential when a user wants to modify only one small part of a complex scene.

In the realm of image restoration, Haoze Sun and the CoSeR team use multi-token embeddings extracted via learnable queries. This approach captures a more nuanced representation of both semantic content and textural appearance than a single global class label, which helps the model avoid "cognitive bias"—the tendency to generate a generic version of an object rather than the specific one present in the low-quality input.

From Pseudo-Words to Full Sentences

The logical conclusion of this trend is the transition from discrete tokens to full, dynamic descriptions. Yang Bai and colleagues demonstrate in Sentence-level Prompts Benefit Composed Image Retrieval that "sentence-level prompts" generated by a vision-language backbone (like BLIP-2) provide significantly more expressivity than pseudo-words. These sentences allow the model to explicitly describe which elements to keep and which to ignore, making it particularly effective for difficult tasks like object removal.

Even in high-speed applications like face generation, the multi-token strategy prevails. Zhuowei Chen shows in DreamIdentity that projecting facial features into multiple distinct pseudo-word embeddings prevents the information bottleneck that usually plagues "zero-shot" identity preservation. This allows for high-fidelity faces that remain highly editable via text prompts.

By breaking the "one image, one word" rule, these researchers have unlocked a more flexible and precise way for machines to understand visual composition, moving us closer to a world where AI can manipulate images with the same descriptive fluidity as human language.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_token_textual_inversion_for_fine_g_aca16189,
  title = {Multi-Token Textual Inversion for Fine-Grained Visual Composition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-token-textual-inversion-for-fine-grained-visual-composition/rdr_aca161893e6342c89f0ab8c7846f4fb6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_token_textual_inversion_for_fine_g_aca16189_v1,
  title = {Multi-Token Textual Inversion for Fine-Grained Visual Composition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-token-textual-inversion-for-fine-grained-visual-composition/rdr_aca161893e6342c89f0ab8c7846f4fb6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}