TL;DR
The authors introduce SEMASIA, a massive, standardized dataset containing the latent representations of nearly 1,700 pretrained vision models evaluated across eight standard image benchmarks. By pairing billions of embedding vectors with rich architectural and training metadata, SEMASIA provides a reproducible foundation for studying latent space geometry and benchmarking alignment methods without requiring expensive GPU infrastructure.
What Problem Does This Paper Address?
Deep neural networks map raw inputs into continuous latent spaces where semantic similarity often translates to geometric proximity. However, these latent spaces are notoriously incompatible across different models. Change the architecture, the pretraining data, the objective function, or even just the random seed, and you get embeddings that capture similar concepts but possess entirely different geometries.
This "latent space alignment problem" is a major roadblock for model interpretability, transfer learning, federated learning, and semantic communication (where models interact directly via embeddings). Until now, progress has been bottlenecked by the lack of a large-scale, model-diverse benchmark to systematically study and align these spaces.
Methods
The authors built SEMASIA by running inference on 1,697 pretrained vision models from the popular timm library. They extracted neural activations from the "semantic bottleneck"—the final layer immediately preceding the classification head.
To make this data actionable, they accompanied the embeddings with a comprehensive Model Registry documenting metadata like model family, depth, width, pretraining datasets, and augmentation strategies.
Using this framework, they demonstrated three applications:
- Conceptual Organization Analysis: Studying how individual latent spaces cluster concepts using t-SNE and UMAP.
- Alignment Benchmarking: Evaluating three supervised latent alignment techniques: a standard Linear map, a prototype-anchor projection (PPFE), and Canonical Correlation Analysis (CCA).
- Large-Scale Regression Analysis: Applying Ordinary Least Squares (OLS) regression to analyze how specific training choices (like data augmentation or model scale) causally impact geometric properties like effective rank and isotropy.
Datasets
The SEMASIA collection standardizes embeddings extracted across eight widely used vision benchmarks:
- CIFAR-10 & CIFAR-100
- MNIST & Fashion-MNIST
- Oxford Flowers
- Tiny ImageNet & ImageNet-1k
- CelebA
The dataset is organized into Parquet files, providing a unique id for each input image to allow exact sample-to-sample correspondence across all 1,697 models.
Key Insights and Ideas
- The "Platonic" Convergence is Real but Granular: Visualizing the latent spaces reveals that models independently discover hierarchical semantic neighborhoods. For example, flower images from completely different datasets collapse into the same geometric regions, showing that models naturally learn prototype-like clusters regardless of their training source.
- Simplicity Wins in Alignment: When mapping one model's latent space to another, a simple linear map consistently outperforms more complex methods like CCA or PPFE. This is especially true under heavy compression (low latent dimensions), where linear maps preserve downstream task accuracy far better.
- Geometric Signatures of Catastrophic Forgetting: The large-scale regression analysis revealed distinct geometric fingerprints for different training regimes. For instance, "Specialization" (fine-tuning a large model on a smaller dataset) compresses the embedding space and reduces its effective rank—acting as a clear geometric signature of catastrophic forgetting.
- Scale vs. Augmentation: Both increasing model scale and adding data augmentation widen the embedding space. However, larger models increase semantic coding capacity while reducing isotropy (variance concentrates along dominant directions), whereas data augmentation merely redistributes variance efficiently within existing dimensions.
Key Results
- Scale of the Dataset: The dataset contains billions of data points (e.g., over 169 million total rows for the ImageNet-1k test split alone), democratizing representation learning research for groups without massive GPU clusters.
- Alignment Performance: In alignment benchmarks, the Linear baseline maintains high downstream accuracy even when compressed down to 16 dimensions, whereas CCA and PPFE performance drops sharply.
- Manifold Topology: Graph-based topological analysis proved that architectural families (e.g., Convolutional vs. Vision Transformers) organize representations using fundamentally different manifold topologies, as verified by significant likelihood-ratio tests ($p < 0.001$).
Limitations and Future Work
- Vision-Only: The current release is strictly limited to image models, preventing cross-modal comparisons (e.g., vision-to-language alignment) which are crucial for testing the Platonic Representation Hypothesis broadly.
- Static Layer and Time: Embeddings are extracted from a single fixed layer at the very end of training. The authors note that future extensions should explore depth (how representations evolve across layers) and time (how geometry emerges across training epochs).
Summary updated