Mapping the Multiverse of Latent Representations

Jeremy Wayland, Corinna Coupette, Bastian Rieck

In the world of machine learning, we’ve become incredibly good at building models that work, but we’re still surprisingly bad at understanding why they work—or why they suddenly fail. We often focus on performance metrics like accuracy or MSE, but these numbers hide a deeper reality: the "latent spaces" these models create are highly sensitive to every choice we make, from hyperparameters to random seeds.

TL;DR

The authors introduce PRESTO, a framework that uses persistent homology (a tool from algebraic topology) to map the "multiverse" of possible latent representations a model can create. By analyzing the topology of these spaces, PRESTO can detect anomalous models, measure hyperparameter sensitivity, and even compress search spaces by 50% without losing structural information.

What Problem Does This Paper Address?

Modern ML relies heavily on latent representations—think of the embeddings in a Large Language Model (LLM) or the bottleneck in a Variational Autoencoder (VAE). However, these representations are notoriously unstable. A tiny tweak in a learning rate or a different random initialization can lead to a radically different latent structure, even if the final performance score remains the same. This is known as representational variability.

This variability contributes to a "reproducibility crisis" in ML. If two models perform equally well but have completely different internal structures, which one is more robust? Which one is more interpretable? The authors argue that we need to move beyond "performance-driven" evaluation and embrace multiverse analysis—explicitly studying the entire range of possible outcomes resulting from different (but reasonable) modeling choices.

Methods

The core of the paper is PRESTO (PRojected Embedding Similarity via Topological Overlays). It treats a collection of latent spaces as a "multiverse" and uses topology to compare them.

The PRESTO pipeline (illustrated in Figure 2) follows these steps:

  1. Embed: Generate latent representations for a dataset.
  2. Project: Scale the high-dimensional embeddings down to a manageable size (e.g., using PCA or random projections). The authors prove that this projection step preserves essential topological features.
  3. Persistent Homology: This is the "secret sauce." It analyzes the data at multiple scales to find persistent features like clusters (0D), loops (1D), or voids (2D).
  4. Persistence Landscapes: These topological features are converted into "landscapes"—vectorized summaries that live in a standard mathematical space (a Banach space), allowing for easy statistical comparison.
Figure 2. The PRESTO pipeline. For each model Mi in our multi- verse M, PRESTO computes the persistent homology associated with the embedding of a dataset Xi generated by Mi, yielding a set of embeddings E. Thus enabled to compare the latent spaces of different models via the landscape distance of their persistence landscapes, with PRESTO, we can cluster, compress, detect outliers in, and analyze the sensitivity of (hyper)parameter configurations.

By comparing these landscapes, PRESTO can calculate a PRESTO Distance (PD) between two models or a PRESTO Variance (PV) for an entire multiverse of models.

Datasets

The authors evaluated PRESTO across a diverse set of multiverses:

  • Image Generation (VAEs): Used celebA, CIFAR-10, dsprites, FashionMNIST, and MNIST. They compared three architectures: $\beta$-VAE, INFOVAE, and WAE.
  • Natural Language (Transformers): Used abstracts from arXiv, bbc, cnn, and patents. They compared six models, ranging from small sentence transformers (like MiniLM) to large language models (like OpenAI's ada and MistralAI's MISTRAL).
  • Dimensionality Reduction: Evaluated standard techniques like UMAP, t-SNE, PHATE, Isomap, and LLE on various datasets.

Key Insights and Ideas

  • Topology is Orthogonal to Performance: A major "aha" moment is that representational variability doesn't always track with performance variability. Two models can have nearly identical test loss but completely different latent topologies. PRESTO reveals structural instabilities that performance metrics miss.
  • Representational Stability as a Goal: The authors suggest that models with lower representational variability (more stable latent spaces) should be preferred, as they are likely more robust and interpretable.
  • Hyperparameter Knowledge is (Sometimes) Reusable: For some models, like $\beta$-VAE, the topological structure is consistent across different datasets. This means you can optimize hyperparameters on a small "proxy" dataset and trust that they will work on your larger target dataset. However, for other models like INFOVAE, this isn't the case.
  • Distinctive Signal: PRESTO captures structural information that existing measures like CKA (Centered Kernel Alignment) or RTD (Representation Topology Divergence) miss, providing a more nuanced view of model similarity (see Figure 3).
Figure 3. Comparing PRESTO distances with other measures. We show the Pearson correlations between PRESTO, RTD, kCKA, and lCKA, on random data (left), VAE embeddings (center), and LLM embeddings (right). PRESTO captures representational variation differently from existing methods.

Key Results

  • Search Space Compression: By identifying models that are topologically similar, PRESTO can compress a hyperparameter search space by 50% while ensuring that the remaining models still represent the full structural diversity of the original multiverse.
  • LLM Discrimination: PRESTO’s topological comparisons clearly distinguish between large language models (LLMs) and smaller transformer models, showing that LLMs create fundamentally different latent structures.
  • Outlier Detection: The framework successfully identifies "anomalous" training runs—configurations that might look okay on paper but have collapsed or distorted latent spaces.
  • Efficiency: Despite using complex topology, the authors demonstrate that PRESTO is highly scalable, running in approximately linear time relative to the number of samples.
PRESTO can halve the size of a hyperparameter search space while maintaining low topological distortion.

Limitations and Future Work

While powerful, PRESTO is still in its early stages. The authors note that:

  • Pipeline Sensitivity: The choice of normalization and projection method (PCA vs. random) affects what PRESTO "sees" (geometric vs. topological variation). More work is needed to provide clear guidelines for these choices.
  • Beyond Generative Models: The current work focuses on VAEs and Transformers. Future work should explore GNNs and the internal layers of deep classifiers.
  • The "Why" Question: PRESTO tells us that a latent space has changed, but it doesn't yet explain why a specific hyperparameter choice led to that specific topological shift.

Overall, PRESTO represents a significant step toward "responsible machine learning" by providing the tools to map the hidden, complex multiverse inside our models.

Summary updated

Abstract

Echoing recent calls to counter reliability and robustness concerns in machine learning via multiverse analysis, we present PRESTO, a principled framework for mapping the multiverse of machine-learning models that rely on latent representations. Although such models enjoy widespread adoption, the variability in their embeddings remains poorly understood, resulting in unnecessary complexity and untrustworthy representations. Our framework uses persistent homology to characterize the latent spaces arising from different combinations of diverse machine-learning methods, (hyper)parameter configurations, and datasets, allowing us to measure their pairwise (dis)similarity and statistically reason about their distributions. As we demonstrate both theoretically and empirically, our pipeline preserves desirable properties of collections of latent representations, and it can be leveraged to perform sensitivity analysis, detect anomalous embeddings, or efficiently and effectively navigate hyperparameter search spaces.
BibTeX
@inproceedings{wayland2024mapping,
  author = {Jeremy Wayland and Corinna Coupette and Bastian Rieck},
  title = {{Mapping the Multiverse of Latent Representations}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=SPBxFwIdMk},
}
Figure