Abstract
Latent text representations exhibit geometric regularities, such as the famous analogy: queen is to king what woman is to man. Such structured semantic relations were not demonstrated on image representations. Recent works aiming at bridging this semantic gap embed images and text into a multimodal space, enabling the transfer of text-defined transformations to the image modality. We introduce the SIMAT dataset to evaluate the task of Image Retrieval with Multimodal queries. SIMAT contains 6k images and 18k textual transformation queries that aim at either replacing scene elements or changing pairwise relationships between scene elements. The goal is to retrieve an image consistent with the (source image, text transformation) query. We use an image/text matching oracle (OSCAR) to assess whether the image transformation is successful. The SIMAT dataset will be publicly available. We use SIMAT to evaluate the geometric properties of multimodal embedding spaces trained with an image/text matching objective, like CLIP. We show that vanilla CLIP embeddings are not very well suited to transform images with delta vectors, but that a simple finetuning on the COCO dataset can bring dramatic improvements. We also study whether it is beneficial to leverage pretrained universal sentence encoders (FastText, LASER and LaBSE).
Introduction
In Natural Language Processing, word embeddings are renowned for their geometric regularities, famously illustrated by analogies like King - Man + Woman = Queen. However, translating this "latent space arithmetic" to the visual domain has proven challenging. Researchers from Meta AI and Sorbonne Université address this by exploring Image Retrieval with Multimodal Queries.
The core challenge lies in combining two distinct inputs—a source image and a text-based modification—into a single search query. For instance, given a photo of a cat on grass and the instruction "cat $\rightarrow$ dog," the system should retrieve a similar photo where the cat is replaced by a dog while preserving the original context (the grass).
The SIMAT Dataset
To evaluate this task, the authors introduced the Semantic IMage Transformation (SIMAT) dataset. Derived from Visual Genome, SIMAT consists of 6,000 images and 18,000 textual transformation queries. These queries are designed to test a model's ability to:
- Replace objects: e.g., "cat $\rightarrow$ dog".
- Change attributes: e.g., "red shirt $\rightarrow$ blue shirt".
- Alter relationships: e.g., "sitting $\rightarrow$ jumping".
The dataset provides a rigorous benchmark for measuring how well multimodal models can navigate semantic changes in a visual context.
Methodology: Multimodal Retrieval via Delta Vectors
The proposed approach relies on the concept of Delta Vectors. To perform a transformation, the system operates within a shared multimodal embedding space.
As shown in the figure above, the process involves:
- Encoding: The source and target words ($w_1, w_2$) are mapped to the embedding space.
- Delta Calculation: A "delta vector" is computed as the difference between the target and source embeddings: $\text{delta} = E(w_2) - E(w_1)$.
- Transformation: This delta is added to the source image embedding $E(I)$ with a scaling factor $\lambda$: $E(I_{new}) = E(I) + \lambda \cdot \text{delta}$.
- Retrieval: The system searches a database for the image whose embedding is closest to the transformed vector $E(I_{new})$.
Training and Optimization
The researchers discovered that "vanilla" CLIP, while excellent at zero-shot matching, is not naturally "arithmetic-friendly." To bridge this "arithmetic gap," they fine-tuned CLIP on the COCO dataset using a contrastive loss.
A critical finding was the impact of the temperature parameter ($\tau$) in the contrastive loss function. By setting $\tau \approx 0.1$ during fine-tuning, the researchers could effectively "linearize" the embedding space, making it significantly more suitable for vector addition. This optimization allows the model to treat retrieval as a form of "editing by proxy," finding existing images that match the desired semantic transformation.
Key Results and Insights
- Performance Leap: Fine-tuning CLIP on COCO led to a massive accuracy boost on the SIMAT benchmark, raising the score from 15.9% (zero-shot) to 48.2%.
- Scaling Factor: When the training temperature is correctly tuned, a scaling factor of $\lambda=1$ for the delta vector proved optimal, matching theoretical expectations for word analogies.
- Encoder Versatility: While CLIP was the top performer, the study showed that universal sentence encoders (like LaBSE or LASER) can also be adapted. However, these models are more sensitive to whether the delta is derived from single words or full sentences, with sentence-based deltas generally providing more stability.
Conclusion
This work demonstrates that multimodal embedding spaces can support complex semantic arithmetic with the right training objectives and hyperparameters. While the current SIMAT benchmark focuses on relatively simple triplet transformations, it paves the way for more nuanced image editing and retrieval. Future research will likely explore leveraging large-scale language models to handle higher-level semantic transformations, moving beyond simple linear arithmetic.
Summary updated