Harnessing the Universal Geometry of Embeddings

Rishi Dev Jha, Collin Zhang, Vitaly Shmatikov, John Xavier Morris

Text embeddings are the foundational infrastructure of modern natural language processing, powering everything from retrieval-augmented generation (RAG) and semantic search to text classification. However, a major interoperability problem exists: different embedding models—such as OpenAI's proprietary models versus open-source BERT variants—map the exact same text into completely different, incompatible vector spaces.

Previously, aligning or translating between two embedding spaces required paired data—a massive dataset of identical texts embedded by both models—or direct access to the encoders themselves. But what if you only have a leaked database of embedding vectors from an unknown model? Can you decipher what those vectors mean?

This paper introduces vec2vec, the first unsupervised method to translate text embeddings between entirely different models without needing paired data, predefined matches, or access to the original encoders. By mapping embeddings into a "universal" latent space, the authors demonstrate that different models learn fundamentally similar geometric representations of semantics. This breakthrough supports a strong version of the Platonic Representation Hypothesis and exposes a critical security flaw in how we treat vector databases.

Unsupervised Translation with vec2vec

To tackle the challenge of unsupervised embedding translation, the authors propose an architecture inspired by unsupervised image translation (like CycleGAN) and cross-lingual word alignment.

Instead of relying on paired matches, vec2vec uses a modular setup. It employs encoder-specific adapters that project embeddings into a shared, universal latent space, alongside a shared backbone that processes them. To train this system without paired data, the authors utilize a combination of specialized loss functions:

  • Adversarial Loss (GAN): A discriminator attempts to distinguish between translated embeddings and real embeddings in the target space, forcing the generator to match the target distribution.
  • Reconstruction Loss: Mapping an embedding into the latent space and back to its original space should yield the original vector.
  • Cycle-Consistency: Translating a vector from Space A to Space B, and then back to Space A, should return it to its starting point.
  • Vector Space Preservation (VSP): The pairwise distances (the geometric structure) between a batch of embeddings must remain consistent before and after translation.
Original embeddings vs. vec2vec latent representations

The Universal Geometry of Semantics

The success of vec2vec provides compelling evidence for the Strong Platonic Representation Hypothesis. The paper proves that neural networks trained on similar modalities (like text), even with vastly different architectures, parameter counts, and training datasets, converge on the same universal semantic geometry.

Embeddings possess a "universal language." Even when two embeddings for the same text have near-zero cosine similarity in their respective native spaces, vec2vec can project them into a shared latent space where they become nearly identical.

Security Implications: Vector Databases Are Not Opaque

A major, immediate implication of this work is in data security. It is a common misconception that vector databases are secure by obscurity—that without the original embedding model, the vectors are meaningless noise.

vec2vec architecture and Enron email recovery

This paper proves otherwise. If a company's vector database is compromised, an attacker does not need the original embedding model. They can use vec2vec to translate the unknown vectors into a known, open-source space (like GTE). From there, they can use off-the-shelf "inversion" models to reconstruct the original private text. In their security evaluation, the authors successfully extracted sensitive entities—including names, dates, and financial data from the Enron Email Corpus—purely from translated vectors.

Key Results

  • High-Fidelity Translation: vec2vec successfully translates embeddings across different model backbones (e.g., from T5-based GTR to BERT-based GTE), achieving cosine similarities up to 0.96 and Top-1 retrieval accuracies near 100%. It significantly outperforms Optimal Transport baselines, which fail on cross-backbone pairs.
  • Extreme Robustness: Translators trained purely on Wikipedia data (Natural Questions) successfully translated out-of-distribution embeddings, including colloquial tweets (TweetTopic) and jargon-heavy medical records (MIMIC-III). This proves the translation relies on universal geometry rather than domain-specific memorization.
  • Information Leakage: Using translated embeddings, the authors performed zero-shot attribute inference (e.g., diagnosing diseases from medical embeddings) and text inversion, highlighting severe privacy risks.
  • Cross-Modal Stitching: In a surprising ablation study, vec2vec translated unimodal text embeddings (Granite) into a multimodal space (CLIP), enabling zero-shot text-to-image retrieval without any multimodal training.

Limitations and Future Work

While groundbreaking, the approach has limitations. Generative Adversarial Networks are notoriously sensitive to weight initialization. Training vec2vec across unrelated model families required running multiple random seeds and selecting the one that converged. Additionally, the training process is computationally heavy, taking up to 7 days per model pair on standard GPUs.

Future work will likely focus on improving the stability of the training regime. The authors also point to scaling this method to more modalities (such as audio and video) and developing specialized inversion models tailored specifically for translated embeddings, which could extract even higher-fidelity text from compromised databases.

Summary updated

Abstract

We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.
BibTeX
@inproceedings{jha2025harnessing,
  author = {Rishi Dev Jha and Collin Zhang and Vitaly Shmatikov and John Xavier Morris},
  title = {{Harnessing the Universal Geometry of Embeddings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=jiCLUPq5xv},
}
Figure