Abstract
Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent works leverage text instructions to allow users to more freely express their search intents. However, they primarily focus on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via foundation models. Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves results comparable with or better than prior best on eight benchmarks of various image retrieval tasks, while maintaining high parameter efficiency with a significantly smaller model size. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens.
Introduction: Beyond Visual Similarity
Traditional image retrieval systems usually focus on finding images that are visually similar to a query. However, human search intent is often more complex. For instance, a user might provide a photo of a landmark like the Burj Al Arab and wish to find "other attractions in this country" or an "interior view of this room."
Existing research into Composed Image Retrieval (CIR) attempts to solve this by adding text modifiers to image queries. However, these models are typically limited to a narrow set of pre-defined relations (like changing the color or texture of a shirt) and rely on small, manually labeled datasets. The challenge is building a model that understands truly "open-ended" instructions and complex semantic relationships without requiring millions of expensive, human-written labels.
Methodology: Mining and Synthesizing Relations
The authors introduce MagicLens, which shifts the paradigm from manual labeling to a massive-scale, self-supervised pipeline.
- Mining Natural Relations: The core insight is that images appearing on the same webpage often share a meaningful, implicit relationship (e.g., a product and its charger, or a landmark and a nearby attraction). The authors mined millions of these pairs from the web.
- Instruction Synthesis: To make these implicit relations explicit, they used a "Foundation Model" pipeline. They first used a Large Multimodal Model (LMM), PaLI, to generate detailed captions and labels for each image. Then, they fed this metadata into a Large Language Model (LLM), PaLM2, to synthesize a natural language instruction that connects the two images.
- Model Architecture: MagicLens uses a dual-encoder setup. It takes a query (image + text) and a target (image) and projects them into a shared embedding space. They utilized CoCa and CLIP as backbones, adding self-attention layers to fuse the image and text modalities effectively.
- Contrastive Learning with a Twist: To ensure the model doesn't ignore the text, they used the query image itself (without the instruction) as a "hard negative" during training. This forces the model to learn that the desired target is different from the query image in exactly the way the instruction specifies.
Key Insights and Ideas
- The Web is a Self-Supervised Goldmine: We don't need humans to write "find the interior of this building" if we can find a webpage that already contains both an exterior and an interior shot and use an LLM to describe the link between them.
- Instruction Diversity Matters: The paper shows that "template-free" instructions generated by LLMs lead to much better generalization than the rigid, template-based instructions used in previous work.
- Beyond Visual Similarity: MagicLens learns to handle complex semantic relations, such as comparing the relative scale or geographical context of different objects.
- Small Models, Big Results: MagicLens-B (267M parameters) often outperforms models like CIReVL (14B+ parameters). This suggests that data quality and diversity are more important for "understanding" instructions than raw model size.
Experimental Results
The model was tested across eight diverse benchmarks, including CIRCO and CIRR (open-domain retrieval), FashionIQ (clothing), and GeneCIS (conditional similarity).
- State-of-the-Art Performance: MagicLens set new records on nearly all benchmarks. On the challenging CIRCO benchmark, MagicLens-L achieved a 34.1 mAP@5, nearly doubling the performance of previous SOTA methods (~18.6).
- Human Preference: In head-to-head human evaluations on "Beyond Visual" queries (requiring geographical or temporal knowledge), MagicLens won 80% of the time compared to the previous best model, LinCIR.
- Efficiency: The model achieves these results while being up to 50x smaller in parameter count than some competing modular frameworks that rely on heavy LLM/LMM inference at runtime.
Conclusion and Future Work
While highly effective, the authors noted that ambiguous instructions (e.g., "3D anatomy") can sometimes lead to valid but non-ideal results. Additionally, web-mined data carries inherent risks of bias that require ongoing mitigation. Future directions include applying the MagicLens data construction "recipe" to tasks like visual question answering (VQA) or building more robust multimodal agents.
Summary updated