The Indra Representation Hypothesis for Multimodal Alignment

Jianglin Lu, Hailing Wang, Kuo Yang, Yitian Zhang, Simon Jenni, Yun Fu

Introduction

Unimodal foundation models—such as standalone vision or text models—naturally learn similar, convergent representations of reality. However, aligning them typically requires expensive retraining and massive paired datasets. Real-world data is inherently multimodal, but foundation models are often trained on a single modality (e.g., BERT for text, ViT for images). To build multimodal systems, researchers usually have to align these models using cross-modal objectives.

Interestingly, recent research suggests that powerful unimodal models actually converge on similar underlying representations of reality anyway. However, we cannot easily use this because raw model outputs (embeddings) are "structurally myopic"—they represent a sample in isolation. Furthermore, embeddings from different models usually have different dimensions (e.g., a 768-D text vector vs. a 1024-D image vector), making direct comparison impossible without a learned translation layer.

This paper tackles a fundamental question: How can we extract and utilize the shared structural reality learned by these models to align them without any retraining? The authors introduce the "Indra Representation," which represents data points not by isolated embeddings, but by their relational distances to all other points in a dataset. This simple, theoretically grounded shift enables robust, training-free alignment across different modalities like vision, language, and audio.

The Indra Representation

The authors propose the Indra Representation, inspired by the Buddhist philosophical metaphor of Indra's Net—a vast cosmic web where every jewel reflects every other jewel. Instead of representing a data point by its internal, isolated embedding, the Indra Representation defines a data point by its relational profile to all other data points.

Key Concepts

  1. The Theory: Grounded in category theory (specifically the V-enriched Yoneda embedding), the authors mathematically prove that an object can be uniquely and completely characterized by its relationships (morphisms) to all other objects. Unlike many heuristic alignment tricks, the Indra Representation is backed by the Yoneda Lemma, proving that this relational profile uniquely and completely characterizes the data while preserving its structural properties.
  2. The Instantiation: In practice, instead of using a model's raw output vector $f(x)$ for a sample, the Indra representation is a vector of angular distances between that sample and every other sample in the dataset: $[d(x, x_1), d(x, x_2), \dots, d(x, x_n)]$.
  3. Cross-Modal Matching: Because these relational profiles capture the fundamental structure of the data, dimensional incompatibility vanishes. A text embedding and an image embedding both become $N$-dimensional relational vectors (where $N$ is the dataset size). You can then directly compare the text's relational vector to the image's relational vector to see if they match.

The true "universal" representation learned by foundation models isn't an absolute coordinate in a latent space, but the web of relationships between data points. By converting embeddings into distance profiles relative to the dataset, the authors completely bypass the dimensional mismatch problem between different architectures.

Experimental Validation

The authors evaluated their method across several benchmarks, spanning single-modality robustness and cross-modal alignment:

  • Single Modality (Vision Robustness): CIFAR-10, CIFAR-100, and Office-Home.
  • Vision-Language Alignment: MS-COCO and NOCAPS (used for zero-shot image captioning and retrieval tasks).
  • Speech-Language Alignment: TIMIT (a dataset of phonetically rich spoken sentences and their text transcripts).

Key Results

Noise Robustness: Indra representations are highly resilient to feature corruption. As shown in the results below, when severe Gaussian noise is injected into the features, the Indra representation maintains significantly higher accuracy. For instance, on CIFAR-100 with a noise level of $\sigma=7.0$, the Indra representation (using a Dinov2 backbone) maintained 58.67% accuracy, compared to a steep drop to 40.16% for the original embeddings.

Accuracy under different Gaussian noise levels

Vision-Language Alignment: Without any cross-modal training, the Indra representation massively improved retrieval between unaligned models. On MS-COCO, using standalone ViT and RoBERTa models, Top-5 Image-to-Text retrieval accuracy jumped from 0.491 (original) to 0.880 (Indra).

Audio-Language Alignment: The method also successfully bridged speech and text. On the TIMIT dataset, Top-5 Text-to-Audio retrieval improved from 0.472 to 0.504 (using wavlm-l and RoBERTa).

Limitations and Future Directions

The primary limitation of the exact Indra Representation is scalability. Because it requires calculating the distance between every pair of samples, it has a computational complexity of $O(n^2d)$ and a memory complexity of $O(n^2)$. This quadratic scaling makes it difficult to apply natively to massive, web-scale datasets.

To address this in future work, the authors suggest using approximation techniques. Instead of measuring the distance to all samples, the representation could be approximated by measuring distances to a small, fixed subset of "landmark" samples, or by utilizing approximate nearest neighbor (ANN) search algorithms to maintain the structural benefits without the massive compute overhead.

Summary updated

Abstract

Recent studies have uncovered an interesting phenomenon: unimodal foundation models tend to learn convergent representations, regardless of differences in architecture, training objectives, or data modalities. However, these representations are essentially internal abstractions of samples that characterize samples independently, leading to limited expressiveness. In this paper, we propose The Indra Representation Hypothesis, inspired by the philosophical metaphor of Indra’s Net. We argue that representations from unimodal foundation models are converging to implicitly reflect a shared relational structure underlying reality, akin to the relational ontology of Indra’s Net. We formalize this hypothesis using the V-enriched Yoneda embedding from category theory, defining the Indra representation as a relational profile of each sample with respect to others. This formulation is shown to be unique, complete, and structure-preserving under a given cost function. We instantiate the Indra representation using angular distance and evaluate it in cross-model and cross-modal scenarios involving vision, language, and audio. Extensive experiments demonstrate that Indra representations consistently enhance robustness and alignment across architectures and modalities, providing a theoretically grounded and practical framework for training-free alignment of unimodal foundation models. Our code is available at https://github.com/Jianglin954/Indra.
BibTeX
@inproceedings{lu2025indra,
  author = {Jianglin Lu and Hailing Wang and Kuo Yang and Yitian Zhang and Simon Jenni and Yun Fu},
  title = {{The Indra Representation Hypothesis for Multimodal Alignment}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=D2NR5Zq6PG},
}
Figure