Introduction
Unimodal foundation models—such as standalone vision or text models—naturally learn similar, convergent representations of reality. However, aligning them typically requires expensive retraining and massive paired datasets. Real-world data is inherently multimodal, but foundation models are often trained on a single modality (e.g., BERT for text, ViT for images). To build multimodal systems, researchers usually have to align these models using cross-modal objectives.
Interestingly, recent research suggests that powerful unimodal models actually converge on similar underlying representations of reality anyway. However, we cannot easily use this because raw model outputs (embeddings) are "structurally myopic"—they represent a sample in isolation. Furthermore, embeddings from different models usually have different dimensions (e.g., a 768-D text vector vs. a 1024-D image vector), making direct comparison impossible without a learned translation layer.
This paper tackles a fundamental question: How can we extract and utilize the shared structural reality learned by these models to align them without any retraining? The authors introduce the "Indra Representation," which represents data points not by isolated embeddings, but by their relational distances to all other points in a dataset. This simple, theoretically grounded shift enables robust, training-free alignment across different modalities like vision, language, and audio.
The Indra Representation
The authors propose the Indra Representation, inspired by the Buddhist philosophical metaphor of Indra's Net—a vast cosmic web where every jewel reflects every other jewel. Instead of representing a data point by its internal, isolated embedding, the Indra Representation defines a data point by its relational profile to all other data points.
Key Concepts
- The Theory: Grounded in category theory (specifically the V-enriched Yoneda embedding), the authors mathematically prove that an object can be uniquely and completely characterized by its relationships (morphisms) to all other objects. Unlike many heuristic alignment tricks, the Indra Representation is backed by the Yoneda Lemma, proving that this relational profile uniquely and completely characterizes the data while preserving its structural properties.
- The Instantiation: In practice, instead of using a model's raw output vector $f(x)$ for a sample, the Indra representation is a vector of angular distances between that sample and every other sample in the dataset: $[d(x, x_1), d(x, x_2), \dots, d(x, x_n)]$.
- Cross-Modal Matching: Because these relational profiles capture the fundamental structure of the data, dimensional incompatibility vanishes. A text embedding and an image embedding both become $N$-dimensional relational vectors (where $N$ is the dataset size). You can then directly compare the text's relational vector to the image's relational vector to see if they match.
The true "universal" representation learned by foundation models isn't an absolute coordinate in a latent space, but the web of relationships between data points. By converting embeddings into distance profiles relative to the dataset, the authors completely bypass the dimensional mismatch problem between different architectures.
Experimental Validation
The authors evaluated their method across several benchmarks, spanning single-modality robustness and cross-modal alignment:
- Single Modality (Vision Robustness): CIFAR-10, CIFAR-100, and Office-Home.
- Vision-Language Alignment: MS-COCO and NOCAPS (used for zero-shot image captioning and retrieval tasks).
- Speech-Language Alignment: TIMIT (a dataset of phonetically rich spoken sentences and their text transcripts).
Key Results
Noise Robustness: Indra representations are highly resilient to feature corruption. As shown in the results below, when severe Gaussian noise is injected into the features, the Indra representation maintains significantly higher accuracy. For instance, on CIFAR-100 with a noise level of $\sigma=7.0$, the Indra representation (using a Dinov2 backbone) maintained 58.67% accuracy, compared to a steep drop to 40.16% for the original embeddings.
Vision-Language Alignment: Without any cross-modal training, the Indra representation massively improved retrieval between unaligned models. On MS-COCO, using standalone ViT and RoBERTa models, Top-5 Image-to-Text retrieval accuracy jumped from 0.491 (original) to 0.880 (Indra).
Audio-Language Alignment: The method also successfully bridged speech and text. On the TIMIT dataset, Top-5 Text-to-Audio retrieval improved from 0.472 to 0.504 (using wavlm-l and RoBERTa).
Limitations and Future Directions
The primary limitation of the exact Indra Representation is scalability. Because it requires calculating the distance between every pair of samples, it has a computational complexity of $O(n^2d)$ and a memory complexity of $O(n^2)$. This quadratic scaling makes it difficult to apply natively to massive, web-scale datasets.
To address this in future work, the authors suggest using approximation techniques. Instead of measuring the distance to all samples, the representation could be approximated by measuring distances to a small, fixed subset of "landmark" samples, or by utilizing approximate nearest neighbor (ANN) search algorithms to maintain the structural benefits without the massive compute overhead.
Summary updated