MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, Ming-Wei Chang

Overview

MagicLens is a novel self-supervised framework for image retrieval that allows users to search using a combination of a query image and open-ended text instructions. By training on 36.7 million web-mined image pairs—where implicit relations were made explicit using Large Language Models (LLMs)—MagicLens achieves state-of-the-art performance across multiple benchmarks while being up to 50x smaller than previous top-tier models.

The Challenge: Beyond Visual Similarity

Traditional image retrieval systems usually focus on visual similarity: if you search with a photo of a cat, you get more photos of cats. However, real-world search intent is often more complex. You might have an image of a landmark and want to see "the view from inside," or a photo of a car and want to find "the 2013 model in blue."

MagicLens vs. Prior SOTA on open-ended instructions

Existing "Composed Image Retrieval" (CIR) models attempt to solve this by combining images with text. However, they are often limited by small, manually curated datasets that only cover a narrow range of pre-defined relations (like changing the color or texture of an object). The authors of MagicLens argue that we need models capable of understanding open-ended instructions that capture rich, semantic relations beyond simple visual tweaks.

Methodology

The core innovation of MagicLens lies in its data construction pipeline and its efficient architecture.

  1. Mining Implicit Relations: The authors realized that images appearing on the same web page often share meaningful, non-visual relationships (e.g., a product and its charger, or a hotel and a nearby attraction).
  2. Synthesizing Instructions: To turn these implicit web relations into training data, they used a pipeline of foundation models. They first extracted metadata (alt-text, captions, and object labels) for image pairs. Then, they used PaLM2 (an LLM) to synthesize a natural language instruction that describes the relationship between the two images. This resulted in a massive dataset of 36.7M (query image, instruction, target image) triplets.
  3. Dual-Encoder Architecture: MagicLens uses a dual-encoder setup. The query (image + text) is processed through vision and language encoders, then fused using several self-attention layers and a multi-head attention pooler to create a single embedding.
  4. The "Hard Negative" Trick: During training, the authors used the query image itself as a "hard negative." This forces the model to actually listen to the text instruction; if it ignores the text and only looks at the image, it will incorrectly retrieve the query image instead of the target.

Datasets and Evaluation

The model was trained on the newly curated dataset of 36.7M web-mined triplets. It was then evaluated on eight diverse benchmarks, including:

  • CIRCO & CIRR: Open-domain natural image retrieval.
  • FashionIQ (FIQ): Domain-specific (clothing) retrieval.
  • DTIN: Domain transfer (e.g., finding a "cartoon" version of a real horse).
  • GeneCIS: Conditional image similarity.
  • ZS-SBIR: Zero-shot sketch-based image retrieval (using the instruction "find a natural image of it").

Key Insights and Results

  • The Web is the Best Labeler: Instead of paying humans to write instructions, we can leverage the inherent structure of the web. The "co-occurrence" of images on a page is a powerful, free signal for semantic relationships.
  • Instruction Following > Visual Matching: By using the query image as a hard negative, the model learns that "similarity" is defined by the instruction, not just the pixels.
  • Small Models, Big Results: MagicLens-B (based on a 267M parameter CoCa model) outperformed CIReVL (a 14.6B parameter model). This suggests that data quality and diversity are more important for instruction following than raw parameter count.
  • Cross-Modality Synergy: Training on this task actually improved the model's performance on standard text-to-image and image-to-text retrieval, suggesting that understanding complex instructions helps the model learn better general-purpose representations.
  • State-of-the-Art Performance: MagicLens set new state-of-the-art results on several benchmarks, most notably on CIRCO (mAP@5 increased from 12.6 to 34.1) and DTIN (R@10 increased from 12.9 to 48.2).
  • Qualitative Success: As seen in the figure above, the model successfully handles complex reasoning, such as finding the "Palm Islands" when given an image of the "Burj Al Arab" and the instruction "Find other attractions in this country." Human evaluators on a 1.4M-image unseen corpus found that MagicLens significantly outperformed the previous best model, especially on "Beyond Visual" instructions (80% win rate).

Limitations and Future Work

While highly capable, MagicLens has limitations. If an instruction is too vague (e.g., "3D rendering of a different body part"), the model might return a valid but unexpected result. Additionally, while filtered, web-mined data can still contain biases or low-quality pairs that might affect model behavior.

Looking forward, the authors suggest that MagicLens could be integrated into larger multimodal agents or used to enhance Retrieval-Augmented Generation (RAG) systems, where precise image pinpointing is required.

Summary updated

Abstract

Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent works leverage text instructions to allow users to more freely express their search intents. However, they primarily focus on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via foundation models. Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves results comparable with or better than prior best on eight benchmarks of various image retrieval tasks, while maintaining high parameter efficiency with a significantly smaller model size. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens. Code and models are publicly available at the https://open-vision-language.github.io/MagicLens/.
BibTeX
@inproceedings{zhang2024magiclens,
  author = {Kai Zhang and Yi Luan and Hexiang Hu and Kenton Lee and Siyuan Qiao and Wenhu Chen and Yu Su and Ming-Wei Chang},
  title = {{MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=Zc22RDtsvP},
}
Figure