In natural language processing, the discovery that word embeddings could support semantic analogies through simple linear algebra—famously, $vec(\text{"king"}) - vec(\text{"man"}) + vec(\text{"woman"}) \approx vec(\text{"queen"})$—revealed that high-dimensional spaces could capture structured human knowledge. A growing body of research extends this intuition to the multimodal domain, demonstrating that when images, text, and other modalities are mapped into a shared latent space, the resulting geometry allows for complex semantic manipulation through vector arithmetic.
The Geometry of Shared Spaces
The foundation of this approach is the creation of a joint embedding space where different data types are co-located based on their semantic content. By training models on massive datasets of image-text pairs, researchers like Jia et al. have shown that these latent spaces exhibit linear algebraic properties. In these spaces, an image of a landmark and the text "Eiffel Tower" are not just close to each other; they exist in a neighborhood where directional offsets correspond to consistent conceptual changes.
This alignment allows visual representations to inherit the compositional structure of language. As Bertasius and Torresani observe, when visual object representations are aligned with contextualized language models, they gain the ability to generalize actions or states across unrelated objects through vector operations. For instance, the "chopped" state learned from a tomato can be mathematically applied to a vector representing an onion.
Semantic Manipulation via Delta Vectors
The most intuitive application of this research is the ability to modify a query or an image by adding or subtracting specific attributes. Rather than searching for a new image from scratch, a user can perform "embedding arithmetic" to refine a result.
Couairon et al. demonstrate this by calculating a "delta vector" between two text embeddings (e.g., $vec(\text{"dog"}) - vec(\text{"cat"})$) and adding it to a source image embedding. This operation shifts the image's representation in the latent space toward the target concept while preserving the original context, such as the background or pose. Similarly, Gordo and Larlus show that users can refine image retrieval by applying text modifiers as vector offsets, significantly improving the precision of semantic searches.
This mechanism is not limited to simple object swaps. Ma et al. use vector differences to transfer fine-grained attributes from raw text descriptions into visual illustrations, effectively "injecting" personality or abstract concepts into a baseline image representation.
Zero-Shot Synthesis and Reasoning
Because these spaces are continuous and semantically structured, they enable the identification of concepts the model has never explicitly seen in a specific combination. Zhao et al. demonstrate that by mapping pixels and WordNet concepts into a joint space, a system can synthesize new categories. For example, the intersection of two concept embeddings can help a model identify a "tricycle" by recognizing it shares properties of both "bicycles" and "vehicles."
This "visual-semantic arithmetic" also enables complex relational reasoning. Tewel et al. show that by performing optimization in the shared space, one can solve visual analogies, such as subtracting a national identity from one leader and adding another to generate a new, logically related caption. Park further bridges this gap by mapping visual features directly into the embedding space of a Large Language Model (LLM), allowing the model to solve relational queries through vector manipulation.
Beyond Vision and Text
The principle of algebraic compositionality is increasingly being applied to broader sets of modalities and complex cultural artifacts. Girdhar et al. introduced a framework that binds six modalities—including audio, thermal, and depth data—into a single space, allowing a visual embedding to be combined with an auditory one to retrieve a specific multimodal concept.
Even the evolution of digital culture can be tracked this way. Qu et al. utilize these spaces to map the "evolutionary tree" of memes, identifying how new variants are created by adding the vector of a base image to the vector of a new cultural symbol.
Why It Works: The Role of Alignment
The effectiveness of these arithmetic operations depends heavily on the quality of the alignment between modalities. Wu et al. argue that using soft labels derived from optimal transport, rather than standard contrastive losses, results in superior algebraic compositionality. This suggests that the "arithmetic-friendliness" of a space is not an accident but a property that can be optimized, ensuring that the distance and direction between vectors truly reflect the logical relationships of the human concepts they represent.
Key Foundational Work
Beyond Instance-Level Image Retrieval: Leveraging Captions to Learn a Global Visual Representation for Semantic Retrieval (2017) — contributed 2 concept elements to this direction.
Open Vocabulary Scene Parsing (2017) — contributed 2 concept elements to this direction.
Spectral Graph-Based Method of Multimodal Word Embedding (2017) — contributed 1 concept elements to this direction.