Harnessing the Universal Geometry of Embeddings

Rishi D. Jha, Collin Zhang, Vitaly Shmatikov, John X. Morris

Abstract

We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

Introduction

Text embeddings serve as the foundational "hidden language" of modern artificial intelligence, underpinning critical systems from semantic search engines to Retrieval-Augmented Generation (RAG) pipelines. Despite their ubiquity, these embeddings are traditionally model-specific; a vector generated by a BERT-based encoder exists in a mathematical manifold entirely distinct from one produced by a T5 or GPT-based model. These "incompatible islands" of data create significant friction for interoperability and model migration. Historically, bridging these spaces required access to the original source text or a parallel "bridge" dataset where identical sequences were encoded by both models.

In this work, we present vec2vec, a framework that challenges the necessity of paired data. By leveraging the "Strong Platonic Representation Hypothesis"—the theory that sufficiently capable models converge toward a shared, universal representation of reality—we demonstrate that it is possible to translate between embedding spaces using only the inherent geometry of the vector clouds themselves.

The vec2vec Framework

The vec2vec architecture is a modular, unsupervised framework designed to map embeddings into a shared latent space and subsequently decode them into a target space. The system relies on several key components to ensure high-fidelity translation without ground-truth pairs:

  • Space-Specific Adapters: These modules project embeddings from disparate models into a common dimensionality, allowing for a unified processing pipeline.
  • Shared Backbone: A central architecture that extracts universal semantic features from the projected vectors.
  • Adversarial Training (GANs): To ensure that translated embeddings are indistinguishable from native embeddings in the target space, we employ discriminators that penalize translations falling outside the target distribution.
  • Cycle Consistency: This loss function enforces that a vector translated from Space A to Space B and back to Space A remains mathematically consistent with its origin.
  • Vector Space Preservation (VSP): A critical innovation of this work, VSP ensures that the relative distances and geometric relationships between points are preserved during the translation process, maintaining the semantic integrity of the data.

Experimental Setup and Datasets

To evaluate the robustness of vec2vec, we conducted experiments across a diverse array of datasets and model families, including BERT, T5, CLIP, and GTE. The datasets were selected to test the limits of the universal geometry hypothesis:

  • Natural Questions (NQ): The primary training set, consisting of 2 million samples.
  • TweetTopic: Used to evaluate performance on colloquial language and non-standard characters like emojis.
  • MIMIC-III: A specialized dataset of medical records used to test the framework's ability to handle domain-specific jargon.
  • Enron Email Corpus: Utilized for security evaluations to determine if sensitive corporate information could be recovered from translated vectors.
  • MS COCO: Employed for multimodal studies involving image-text retrieval.

Key Insights: The Universality of Semantics

Our findings provide strong empirical support for the Strong Platonic Representation Hypothesis. We observed that models trained on entirely different datasets with varying architectures (e.g., the multimodal CLIP vs. the text-centric GTE) still converge toward a remarkably similar geometric structure. This suggests that semantics is an inherent property of the data itself rather than a subjective artifact of a specific model's training process.

Furthermore, we demonstrated that unsupervised alignment is not only possible but highly effective. By understanding the "shape" of two independent vector databases, vec2vec can align them without ever seeing the underlying raw text.

Security Implications: The Vulnerability of Geometry

The discovery of a universal embedding geometry introduces a significant security concern. Many organizations treat embedding vectors as "anonymized" versions of sensitive data. However, our research shows that this is a fallacy. If an adversary gains access to a vector database, they can use vec2vec to translate those "unknown" vectors into a "known" space (such as a public BERT model). Once translated, off-the-shelf inversion tools can be used to reconstruct the original sensitive text.

In our tests using the Enron email corpus, we were able to recover sensitive attributes—including names, dates, and specific topics—from 80% of the emails using only translated embeddings and a zero-shot inverter. This highlights that embeddings must be protected with the same rigor as the raw text they represent.

Results and Performance

The vec2vec framework achieved high-fidelity translations, reaching cosine similarities as high as 0.96 between translated vectors and ground-truth targets. Notably, our unsupervised method consistently outperformed "Oracle-aided optimal transport," a strong baseline that utilizes more information than vec2vec.

We also observed success in cross-modality "stitching." For example, we successfully translated unimodal Granite embeddings into the multimodal CLIP space, enabling effective image retrieval from text-only inputs.

Limitations and Future Directions

While promising, the current approach faces challenges. The use of GANs introduces training instability, often requiring multiple initializations to find the optimal model. Additionally, translating to very large models, such as the 4-billion parameter Qwen, is computationally intensive and requires significant convergence time. While the translations preserve enough semantic information for attribute inference, the literal, word-for-word reconstruction of text via inversion remains an imperfect science.

Future work will focus on developing more stable learning algorithms and scaling the framework to even more diverse modalities, including audio and video. The ultimate goal is the creation of a "Universal Translator" for AI representations, enabling seamless communication between any two models in the AI ecosystem.

Summary updated

Abstract

We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.
BibTeX
@inproceedings{jha2025harnessing,
  author = {Rishi D. Jha and Collin Zhang and Vitaly Shmatikov and John X. Morris},
  title = {{Harnessing the Universal Geometry of Embeddings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=jiCLUPq5xv},
}
Figure