In traditional machine learning, images and text were treated as fundamentally different species of data. However, a powerful paradigm has emerged that treats them as different dialects of the same underlying semantic language. By mapping diverse modalities—images, text, audio, and even 3D point clouds—into a single, shared high-dimensional vector space, researchers have discovered a remarkable property: the geometry of this space is semantically meaningful. In these "joint embedding spaces," complex reasoning can be performed using simple linear algebra.
The Manifold of Shared Meaning
The core intuition behind this research direction is that if you train a model to align different modalities—for instance, by ensuring that the vector for the word "dog" is close to the vector for a picture of a dog—the model begins to organize the entire space based on concepts rather than raw pixels or characters.
As Chao Jia and colleagues demonstrated with ALIGN, scaling this alignment to billions of image-text pairs creates a "semantic manifold." In this space, the distance and direction between vectors represent conceptual relationships. This isn't limited to just two modalities. Rohit Girdhar and the Meta AI team showed in ImageBind that images can act as a "binding" bridge to align six different modalities, including thermal and audio data, into one unified representation.
Vector Arithmetic as Conceptual Reasoning
Once different modalities share a coordinate system, we can manipulate them using vector arithmetic. If the space is truly linear, the "difference" between two vectors should represent a specific attribute or transformation.
Consider the classic word embedding example: $King - Man + Woman = Queen$. In multimodal spaces, this logic extends across formats. Yoad Tewel and Yoav Shalev demonstrated this in ZeroCap with "visual-semantic arithmetic." By taking the image embedding of Barack Obama, subtracting the "USA" text vector, and adding a "Germany" text vector, the model shifts the representation toward Angela Merkel.
This works because the model learns that the vector direction from "USA" to "Germany" represents the abstract concept of "nationality shift." Applying that same direction to a person’s image vector moves it to a new point in the space that corresponds to a different person who shares the same relationship to the new country.
Compositional Retrieval and Editing
This geometric regularity enables "Composed Multi-modal Retrieval" (CMR). As explored in the survey by Yongdong Zhang and others, Composed Multi-modal Retrieval: A Survey of Approaches and Applications, a user can query a system with an image of a dress and the text "but shorter and in blue." The system simply adds the "blue" and "short" vectors to the original image vector to find the nearest neighbor in the database.
This principle also powers zero-shot image editing. In FlexIT, Asya Grechka shows that we can transform an image (e.g., "tiger" to "white wolf") by calculating a target point in the latent space: $Target = Image_{source} - Text_{source} + Text_{target}$. By moving the image's representation along this text-defined vector, we can guide a generative model to synthesize a new version of the image that reflects the change.
Interpolation and the Difference Token
While simple addition and subtraction are powerful, researchers are finding that the "path" between two points in the embedding space also contains rich information.
In Difference Inversion, Hyunsoo Kim isolates the mathematical "difference" between two images to create "Difference Tokens." These tokens capture a transformation (like "turning into a sketch") that can then be applied to entirely new images. Similarly, Wenhao Qian and colleagues use spherical linear interpolation in CDGLT to bridge the gap between literal visual meanings and figurative metaphors in memes.
Why It Works: Emergent Alignment
The reason these operations work is "emergent alignment." When a model like OmniBind (developed by Minjie Hong and team) binds 14 different specialist models into one space, it forces them to find a common denominator for concepts.
If a 3D point cloud of a chair, the sound of someone sitting down, and the word "furniture" all map to the same neighborhood, the model has effectively learned a "universal concept" that transcends any single modality. In this unified space, the latent vectors become composable building blocks, allowing us to build, modify, and navigate complex semantic ideas using the simple, elegant language of vectors.