Mapping the Multiverse of Latent Representations

Jeremy Wayland, Corinna Coupette, Bastian Rieck

Abstract

Modern machine learning models often exhibit significant variability in their internal representations, even when achieving similar performance metrics. This "representational variability" poses challenges for reproducibility and model understanding. We present PRESTO (PRojected Embedding Similarity via Topological Overlays), a framework that leverages persistent homology to characterize the topological structure of latent spaces. By treating the collection of potential model configurations as a "multiverse," PRESTO enables the detection of anomalous models, sensitivity analysis, and a 50% reduction in hyperparameter search spaces without compromising structural integrity.

1. Introduction

The "reproducibility crisis" in machine learning is often framed through the lens of performance: can we replicate the accuracy of a reported model? However, a deeper issue lies in the latent space. Two models trained on identical data with minor variations in hyperparameters or random seeds may yield identical loss values while possessing fundamentally different internal geometries. This phenomenon, known as representational variability, suggests that performance metrics alone are insufficient for evaluating model robustness or trustworthiness.

To address this, we propose a "multiverse analysis" approach. Rather than evaluating a single "best" model, we must account for the variation across all reasonable choices in architecture, data preprocessing, and hyperparameters. We introduce PRESTO, a framework designed to map this multiverse using the tools of algebraic topology.

2. The PRESTO Framework

PRESTO provides a mathematically grounded pipeline for comparing latent representations across different models. The process consists of four primary stages:

  1. Data Embedding: High-dimensional latent representations are generated from the target models.
  2. Dimensionality Projection: To maintain computational efficiency, embeddings are projected into a lower-dimensional space ($k \ll d$) using Principal Component Analysis (PCA) or random projections. The framework includes theoretical stability guarantees (Theorem 3.9) ensuring that topological signals are preserved during this reduction.
  3. Persistent Homology (PH): Using $\alpha$-complexes, the framework tracks topological features—such as clusters (0-dimensional) and holes (1-dimensional)—across multiple spatial scales.
  4. Persistence Landscapes: These topological summaries are converted into functions within a Banach space. This conversion allows for rigorous statistical comparison, averaging, and distance measurements between different latent spaces.

3. Experimental Multiverses

The utility of PRESTO was validated across two distinct model multiverses:

  • The VAE Multiverse: An analysis of Variational Autoencoders ($\beta$-VAE, InfoVAE, and WAE) trained on standard image datasets including celebA, CIFAR-10, dsprites, FashionMNIST, and MNIST.
  • The Transformer Multiverse: An evaluation of six pretrained language models, ranging from large-scale models like OpenAI’s Ada and Mistral to smaller sentence-transformers like MiniLM and MPNet. These were tested using text abstracts from arXiv, BBC, CNN, and patent databases.

4. Key Insights

4.1 Topology vs. Performance

A central finding is that representational variability is largely orthogonal to performance variability. Models with nearly identical reconstruction errors often exhibit vastly different latent topologies. This implies that traditional evaluation metrics fail to capture the structural "understanding" a model has developed.

4.2 Hyperparameter Sensitivity

In Disentangling Auto-Encoders (DAEs), PRESTO revealed that true feature disentanglement (e.g., separating color from shape) occurs only within a very narrow range of the hyperparameter space. PRESTO allows researchers to identify these "topologically stable" regions, which are often missed by coarse-grained performance sweeps.

4.3 Signatures of Scale

PRESTO successfully identified distinct topological signatures associated with model scale. Large Language Models (LLMs) demonstrated structural patterns that remained consistent across different text datasets, distinguishing them from smaller models based solely on the geometry of their embeddings.

5. Practical Utility

5.1 Search Space Compression

By identifying "topological representatives" within a multiverse, PRESTO can significantly optimize model development. The authors demonstrated that the hyperparameter search space could be compressed by up to 50% while ensuring that the omitted configurations were topologically redundant to those retained.

5.2 Outlier and Anomaly Detection

By mapping latent spaces into a metric space, PRESTO can identify "anomalous" models—those that fail to converge to a stable representation. These outliers are often undetectable through training loss alone, providing a new diagnostic tool for model training.

6. Conclusion and Future Work

PRESTO bridges the gap between abstract topology and practical machine learning, offering a robust lens through which to view the latent multiverse. While the framework currently focuses on generative models and transformers, future work aims to extend these methods to graph embeddings and internal neural layers. Additionally, the topological alignment of latent spaces presents a promising path toward "zero-shot model stitching," where disparate models can be combined without extensive retraining.

Summary updated

Abstract

Echoing recent calls to counter reliability and robustness concerns in machine learning via *multiverse analysis*, we present PRESTO, a principled framework for *mapping the multiverse* of machine-learning models that rely on *latent representations*. Although such models enjoy widespread adoption, the variability in their embeddings remains poorly understood, resulting in unnecessary complexity and untrustworthy representations. Our framework uses *persistent homology* to characterize the latent spaces arising from different combinations of diverse machine-learning methods, (hyper)parameter configurations, and datasets, allowing us to measure their pairwise *(dis)similarity* and statistically reason about their *distributions*. As we demonstrate both theoretically and empirically, our pipeline preserves desirable properties of collections of latent representations, and it can be leveraged to perform sensitivity analysis, detect anomalous embeddings, or efficiently and effectively navigate hyperparameter search spaces.
BibTeX
@inproceedings{wayland2024mapping,
  author = {Jeremy Wayland and Corinna Coupette and Bastian Rieck},
  title = {{Mapping the Multiverse of Latent Representations}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=SPBxFwIdMk},
}
Figure