Overcoming the Triplet Annotation Bottleneck in Composed Retrieval

66 concepts58 papers3 proposals

Composed Image Retrieval (CIR) allows users to search for a target image using a multimodal query: a reference image combined with a text modifier (e.g., "this dress, but with floral patterns"). While intuitive for users, this task presents a significant challenge for machine learning: the triplet annotation bottleneck.

Standard vision-language tasks like image captioning rely on pairs of data. In contrast, supervised CIR requires triplets consisting of a reference image, a specific text instruction, and a ground-truth target image. As Song et al. note in their comprehensive survey, manually annotating these relational triplets is labor-intensive, expensive, and difficult to scale. This has led to a "data starvation" problem, where specialized CIR datasets remain orders of magnitude smaller than general-purpose vision-language datasets Zhang et al..

To bypass this bottleneck, researchers are shifting away from manual annotation toward three primary strategies: synthetic data generation, zero-shot mapping, and generative imagination.

If human-annotated triplets are too expensive to produce, one solution is to use Large Language Models (LLMs) and Multimodal Models (LMMs) to "hallucinate" the necessary relationships from existing, abundant data.

Ventura et al. (CoVR) demonstrated this by using LLMs to automatically generate 1.6 million training triplets from web video captions, bypassing manual labor entirely. Similarly, Jang et al. introduced the Visual Delta Generator (VDG), which uses an LMM to describe the differences between two images, creating a semi-supervised pipeline that achieves state-of-the-art results. The scale of this approach reached a new peak with MegaPairs, where Zhou et al. synthesized 26 million multimodal triplets to train a universal retriever.

Images as Pseudo-Words

A second strategy, known as Zero-Shot Composed Image Retrieval (ZS-CIR), avoids triplets by leveraging the pre-aligned latent space of models like CLIP. The core intuition is to treat the reference image not as a visual file, but as a "pseudo-word" or token that can be composed with text.

Saito et al. (Pic2Word) pioneered this by training a lightweight network to map image embeddings into the text token space. This allows a user to query the system as if the image were just another word in a sentence. Agnolucci et al. (iSEARLE) refined this through a teacher-student distillation process, while Lin et al. (FTI4CIR) decomposed images into multiple attribute-oriented tokens to capture fine-grained details like sleeve length or texture. By operating entirely in the text-embedding space, these models require no triplet data for training.

Imagining the Target

The most recent evolution in the field leverages the generative power of diffusion models. Instead of trying to map a query to a target, these methods "imagine" what the target should look like.

Wang et al. (CIG) use latent diffusion to generate a pseudo-target image based on the query. This bridges the representation gap: the system retrieves images by comparing the "imagined" target to the actual database, rather than trying to fuse text and image features into a single vector. Gu et al. (CompoDiff) further extended this by using a massive synthetic dataset of 18.8 million triplets to fine-tune this generative process, allowing for controllable search results using spatial masks and negative text.

Beyond General Images

The triplet bottleneck is even more acute in specialized domains. In remote sensing, Ypsilantis et al. introduced WEICOM, a training-free method for satellite imagery retrieval, because high-quality supervised triplets for geographic changes are nearly non-existent. In fashion, where styles evolve rapidly, Mirchandani et al. (FaD-VLP) utilized "pseudo-triplets" describing differences between similar items to train models that can handle relative queries like "more sporty" or "longer sleeves."

By treating the lack of triplet data not as a missing resource, but as a prompt for synthetic and zero-shot innovation, these researchers have transformed CIR from a data-constrained niche into a scalable frontier of multimodal search.

Key Foundational Work

Corpus-Guided Sentence Generation of Natural Images (2011) — contributed 1 concept elements to this direction.
Learning Disentangled Representations of Texts with Application to Biomedical Abstracts (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_triplet_data_scarcity_triplet_annotation_997d9eac,
  title = {triplet data scarcity (triplet annotation bottleneck)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/triplet-data-scarcity-triplet-annotation-bottleneck/rdr_997d9eacaa494d1e8dd5b7bf43a5365d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_triplet_data_scarcity_triplet_annotation_997d9eac_v1,
  title = {triplet data scarcity (triplet annotation bottleneck)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/triplet-data-scarcity-triplet-annotation-bottleneck/rdr_997d9eacaa494d1e8dd5b7bf43a5365d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake