Abstract: Latent text representations exhibit geometric regularities, such as the famous analogy: queen is to king what woman is to man. Such structured semantic relations were not demonstrated on image representations. Recent works aiming at bridging this semantic gap embed images and text into a multimodal space, enabling the transfer of text-defined transformations to the image modality. We introduce the SIMAT dataset to evaluate the task of Image Retrieval with Multimodal queries. SIMAT contains 6k images and 18k textual transformation queries that aim at either replacing scene elements or changing pairwise relationships between scene elements. The goal is to retrieve an image consistent with the (source image, text transformation) query. We use an image/text matching oracle (OSCAR) to assess whether the image transformation is successful. The SIMAT dataset will be publicly available. We use SIMAT to evaluate the geometric properties of multimodal embedding spaces trained with an image/text matching objective, like CLIP. We show that vanilla CLIP embeddings are not very well suited to transform images with delta vectors, but that a simple finetuning on the COCO dataset can bring dramatic improvements. We also study whether it is beneficial to leverage pretrained universal sentence encoders (FastText, LASER and LaBSE).
Introduction
In natural language processing, word embeddings are renowned for their geometric regularities, exemplified by the classic analogy: "king - man + woman = queen." This research investigates whether similar structured semantic relations exist within multimodal embedding spaces, where images and text are co-located. Specifically, the authors address the task of Image Retrieval with Multimodal Queries: finding a target image by applying a text-based transformation to a source image (e.g., "change the cat to a dog").
Methodology: Embedding Arithmetic
The core approach involves "embedding arithmetic" using delta vectors. To transform an image $I$ based on a textual change from $w_1$ to $w_2$, the system follows a three-step process:
- Encoding the Transformation: The difference between the text embeddings of the target and source words is calculated as $\Delta = E_{txt}(w_2) - E_{txt}(w_1)$.
- Applying the Delta: This delta vector is added to the image's embedding, scaled by a factor $\lambda$: $x = E_{img}(I) + \lambda \cdot \Delta$.
- Retrieval: The final embedding $x$ is used to query an image database to find the closest match.
To improve performance, the authors fine-tune CLIP on the COCO dataset. A key finding was the importance of the temperature parameter ($\tau$) in the contrastive loss; a slightly higher $\tau \approx 0.1$ (compared to standard matching tasks) promotes a more linear embedding space, making it significantly more conducive to vector addition. To automate the evaluation process, the authors employ OSCAR (Object-Semantics Aligned Pre-training) as an oracle, assessing whether the retrieved image matches the intended target caption.
The SIMAT Dataset
To evaluate this task, the authors introduced the Semantic IMage Transformation (SIMAT) dataset. SIMAT consists of 5,989 images sourced from Visual Genome, paired with 17,996 transformation queries. These queries are designed to test a model's ability to handle various semantic shifts, including subject swaps, object replacements, and relationship changes.
Results and Key Insights
The study reveals that "vanilla" zero-shot CLIP is poorly suited for embedding arithmetic, achieving a SIMAT score of only 15.9%. However, fine-tuning with optimized hyperparameters leads to a dramatic improvement, reaching 48.2%—a 33% absolute gain. This fine-tuned "delta vector" approach outperformed a baseline "Image $\rightarrow$ Text $\rightarrow$ Image" method (39.7%), confirming that direct manipulation of embeddings is more effective than intermediate translation to natural language.
Several critical insights emerged:
- Retrieval vs. Transformation: Success in standard image-text retrieval does not guarantee success in image transformation. These tasks rely on different geometric properties of the latent space.
- Linearity and Scaling: In a perfectly linear space, the ideal scaling factor $\lambda$ should be 1. While vanilla CLIP requires high $\lambda$ values to register changes, the fine-tuned model achieves peak performance at $\lambda=1$.
- Sentence Encoders: While single-word deltas fail for specialized sentence encoders like LASER or LaBSE, they become highly effective when deltas are averaged across multiple descriptive sentences (e.g., "A cat on grass" $\rightarrow$ "A dog on grass").
Conclusion
This work demonstrates that while multimodal spaces do not inherently possess the same arithmetic properties as word embeddings, they can be conditioned to do so through targeted fine-tuning. The SIMAT dataset provides a rigorous benchmark for future research in this area. Future directions include expanding to more abstract transformations and integrating large-scale language models to handle complex semantic nuances.
Summary updated