In the world of machine learning, we’ve become incredibly good at building models that work, but we’re still surprisingly bad at understanding why they work—or why they suddenly fail. We often focus on performance metrics like accuracy or MSE, but these numbers hide a deeper reality: the "latent spaces" these models create are highly sensitive to every choice we make, from hyperparameters to random seeds.
TL;DR
The authors introduce PRESTO, a framework that uses persistent homology (a tool from algebraic topology) to map the "multiverse" of possible latent representations a model can create. By analyzing the topology of these spaces, PRESTO can detect anomalous models, measure hyperparameter sensitivity, and even compress search spaces by 50% without losing structural information.
What Problem Does This Paper Address?
Modern ML relies heavily on latent representations—think of the embeddings in a Large Language Model (LLM) or the bottleneck in a Variational Autoencoder (VAE). However, these representations are notoriously unstable. A tiny tweak in a learning rate or a different random initialization can lead to a radically different latent structure, even if the final performance score remains the same. This is known as representational variability.
This variability contributes to a "reproducibility crisis" in ML. If two models perform equally well but have completely different internal structures, which one is more robust? Which one is more interpretable? The authors argue that we need to move beyond "performance-driven" evaluation and embrace multiverse analysis—explicitly studying the entire range of possible outcomes resulting from different (but reasonable) modeling choices.
Methods
The core of the paper is PRESTO (PRojected Embedding Similarity via Topological Overlays). It treats a collection of latent spaces as a "multiverse" and uses topology to compare them.
The PRESTO pipeline (illustrated in Figure 2) follows these steps:
- Embed: Generate latent representations for a dataset.
- Project: Scale the high-dimensional embeddings down to a manageable size (e.g., using PCA or random projections). The authors prove that this projection step preserves essential topological features.
- Persistent Homology: This is the "secret sauce." It analyzes the data at multiple scales to find persistent features like clusters (0D), loops (1D), or voids (2D).
- Persistence Landscapes: These topological features are converted into "landscapes"—vectorized summaries that live in a standard mathematical space (a Banach space), allowing for easy statistical comparison.
By comparing these landscapes, PRESTO can calculate a PRESTO Distance (PD) between two models or a PRESTO Variance (PV) for an entire multiverse of models.
Datasets
The authors evaluated PRESTO across a diverse set of multiverses:
- Image Generation (VAEs): Used
celebA,CIFAR-10,dsprites,FashionMNIST, andMNIST. They compared three architectures: $\beta$-VAE, INFOVAE, and WAE. - Natural Language (Transformers): Used abstracts from
arXiv,bbc,cnn, andpatents. They compared six models, ranging from small sentence transformers (likeMiniLM) to large language models (like OpenAI'sadaand MistralAI'sMISTRAL). - Dimensionality Reduction: Evaluated standard techniques like
UMAP,t-SNE,PHATE,Isomap, andLLEon various datasets.
Key Insights and Ideas
- Topology is Orthogonal to Performance: A major "aha" moment is that representational variability doesn't always track with performance variability. Two models can have nearly identical test loss but completely different latent topologies. PRESTO reveals structural instabilities that performance metrics miss.
- Representational Stability as a Goal: The authors suggest that models with lower representational variability (more stable latent spaces) should be preferred, as they are likely more robust and interpretable.
- Hyperparameter Knowledge is (Sometimes) Reusable: For some models, like $\beta$-VAE, the topological structure is consistent across different datasets. This means you can optimize hyperparameters on a small "proxy" dataset and trust that they will work on your larger target dataset. However, for other models like INFOVAE, this isn't the case.
- Distinctive Signal: PRESTO captures structural information that existing measures like CKA (Centered Kernel Alignment) or RTD (Representation Topology Divergence) miss, providing a more nuanced view of model similarity (see Figure 3).
Key Results
- Search Space Compression: By identifying models that are topologically similar, PRESTO can compress a hyperparameter search space by 50% while ensuring that the remaining models still represent the full structural diversity of the original multiverse.
- LLM Discrimination: PRESTO’s topological comparisons clearly distinguish between large language models (LLMs) and smaller transformer models, showing that LLMs create fundamentally different latent structures.
- Outlier Detection: The framework successfully identifies "anomalous" training runs—configurations that might look okay on paper but have collapsed or distorted latent spaces.
- Efficiency: Despite using complex topology, the authors demonstrate that PRESTO is highly scalable, running in approximately linear time relative to the number of samples.
Limitations and Future Work
While powerful, PRESTO is still in its early stages. The authors note that:
- Pipeline Sensitivity: The choice of normalization and projection method (PCA vs. random) affects what PRESTO "sees" (geometric vs. topological variation). More work is needed to provide clear guidelines for these choices.
- Beyond Generative Models: The current work focuses on VAEs and Transformers. Future work should explore GNNs and the internal layers of deep classifiers.
- The "Why" Question: PRESTO tells us that a latent space has changed, but it doesn't yet explain why a specific hyperparameter choice led to that specific topological shift.
Overall, PRESTO represents a significant step toward "responsible machine learning" by providing the tools to map the hidden, complex multiverse inside our models.
Summary updated