SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations

Mario Edoardo Pandolfo, Enrico Grimaldi, Lorenzo Marinucci, Leonardo Di Nino, Simone Fiorellino, Sergio Barbarossa, Paolo Di LorenzoarXiv 2026 · arXiv

TL;DR

The authors introduce SEMASIA, a massive, standardized dataset containing the latent representations of nearly 1,700 pretrained vision models evaluated across eight standard image benchmarks. By pairing billions of embedding vectors with rich architectural and training metadata, SEMASIA provides a reproducible foundation for studying latent space geometry and benchmarking alignment methods without requiring expensive GPU infrastructure.

What Problem Does This Paper Address?

Deep neural networks map raw inputs into continuous latent spaces where semantic similarity often translates to geometric proximity. However, these latent spaces are notoriously incompatible across different models. Change the architecture, the pretraining data, the objective function, or even just the random seed, and you get embeddings that capture similar concepts but possess entirely different geometries.

This "latent space alignment problem" is a major roadblock for model interpretability, transfer learning, federated learning, and semantic communication (where models interact directly via embeddings). Until now, progress has been bottlenecked by the lack of a large-scale, model-diverse benchmark to systematically study and align these spaces.

Methods

The authors built SEMASIA by running inference on 1,697 pretrained vision models from the popular timm library. They extracted neural activations from the "semantic bottleneck"—the final layer immediately preceding the classification head.

To make this data actionable, they accompanied the embeddings with a comprehensive Model Registry documenting metadata like model family, depth, width, pretraining datasets, and augmentation strategies.

Using this framework, they demonstrated three applications:

  1. Conceptual Organization Analysis: Studying how individual latent spaces cluster concepts using t-SNE and UMAP.
  2. Alignment Benchmarking: Evaluating three supervised latent alignment techniques: a standard Linear map, a prototype-anchor projection (PPFE), and Canonical Correlation Analysis (CCA).
  3. Large-Scale Regression Analysis: Applying Ordinary Least Squares (OLS) regression to analyze how specific training choices (like data augmentation or model scale) causally impact geometric properties like effective rank and isotropy.

Datasets

The SEMASIA collection standardizes embeddings extracted across eight widely used vision benchmarks:

  • CIFAR-10 & CIFAR-100
  • MNIST & Fashion-MNIST
  • Oxford Flowers
  • Tiny ImageNet & ImageNet-1k
  • CelebA

The dataset is organized into Parquet files, providing a unique id for each input image to allow exact sample-to-sample correspondence across all 1,697 models.

Key Insights and Ideas

  • The "Platonic" Convergence is Real but Granular: Visualizing the latent spaces reveals that models independently discover hierarchical semantic neighborhoods. For example, flower images from completely different datasets collapse into the same geometric regions, showing that models naturally learn prototype-like clusters regardless of their training source.
Figure 2: Two-dimensional t-SNE projection of the aimv2_1b_patch14_224.apple_pt 2048- dimensional latent space, populated with samples from seven SEMASIA benchmarks. Each benchmark forms its own cluster, but semantically overlapping concepts collapse onto shared neighborhoods regardless of source: flower images from Oxford Flowers and CIFAR-100 occupy the same region, as do large mammals and vehicles drawn from CIFAR-10, CIFAR-100, and Tiny-ImageNet (highlighted with the related origin dataset color).
  • Simplicity Wins in Alignment: When mapping one model's latent space to another, a simple linear map consistently outperforms more complex methods like CCA or PPFE. This is especially true under heavy compression (low latent dimensions), where linear maps preserve downstream task accuracy far better.
  • Geometric Signatures of Catastrophic Forgetting: The large-scale regression analysis revealed distinct geometric fingerprints for different training regimes. For instance, "Specialization" (fine-tuning a large model on a smaller dataset) compresses the embedding space and reduces its effective rank—acting as a clear geometric signature of catastrophic forgetting.
  • Scale vs. Augmentation: Both increasing model scale and adding data augmentation widen the embedding space. However, larger models increase semantic coding capacity while reducing isotropy (variance concentrates along dominant directions), whereas data augmentation merely redistributes variance efficiently within existing dimensions.
Figure 5: Forest plot of pooled OLS regression coefficients ˆβ for the embedding geometry and linear probing metrics across five pretraining conditions (see Table 2 for condition definitions). Probing scores (accuracy, recall, precision, F1) are obtained via a least-squares linear probe. All ˆβ are expressed in units of the control-group standard deviation of each metric (σcontrol, pooled across datasets and analysis types), yielding an effect-size measure analogous to Cohen’s d. Observations are pooled across CIFAR-10, MNIST, Fashion-MNIST, and Oxford Flowers.

Key Results

  • Scale of the Dataset: The dataset contains billions of data points (e.g., over 169 million total rows for the ImageNet-1k test split alone), democratizing representation learning research for groups without massive GPU clusters.
  • Alignment Performance: In alignment benchmarks, the Linear baseline maintains high downstream accuracy even when compressed down to 16 dimensions, whereas CCA and PPFE performance drops sharply.
Figure 4: Comparison of three supervised alignment methods on every model pair from Figure 12: LINEAR (the Eigen-K baseline from 57), PPFE [53], and Canonical Correlation Analysis (CCA) [14]. The x-axis reports the number of non-zero components K retained by the alignment map (i.e., the active latent dimensions); the y-axis reports downstream task accuracy after communication (top) and mean squared error of latent reconstruction (bottom). Method specifications and metric definitions are detailed in Appendix E.
  • Manifold Topology: Graph-based topological analysis proved that architectural families (e.g., Convolutional vs. Vision Transformers) organize representations using fundamentally different manifold topologies, as verified by significant likelihood-ratio tests ($p < 0.001$).

Limitations and Future Work

  • Vision-Only: The current release is strictly limited to image models, preventing cross-modal comparisons (e.g., vision-to-language alignment) which are crucial for testing the Platonic Representation Hypothesis broadly.
  • Static Layer and Time: Embeddings are extracted from a single fixed layer at the very end of training. The authors note that future extensions should explore depth (how representations evolve across layers) and time (how geometry emerges across training epochs).

Summary updated

Abstract

Latent representations learned by neural networks often exhibit semantic structure, where concept similarity is reflected by geometric proximity in embedding space. However, comparing such spaces across models remains difficult: changes in architecture, pretraining data, objective, or random seed can yield embeddings with similar content but incompatible geometry. This latent space alignment problem is central to interpretability, transfer and multimodal learning, federated systems, and semantic communication; however, progress remains limited by the lack of large-scale, model-diverse, and metadata-rich benchmarks. To address this gap, we introduce SEMASIA, a large-scale collection of latent representations extracted from approximately 1,700 pretrained vision models across eight standard image-classification benchmarks. SEMASIA pairs embeddings with structured metadata describing architectures, training regimes, pretraining sources, and model scale. We demonstrate three applications of the resource. First, we analyze the conceptual organization of individual latent spaces, showing consistent prototype-like clustering and hierarchical semantic neighborhoods across models and datasets. Second, we benchmark supervised alignment mappings between latent spaces using reconstruction error and downstream task performance. Third, we perform a large-scale regression analysis of how pretraining-data complexity, specialization, transfer learning, augmentation, and model scale relate to geometric and probing properties of embeddings. By coupling representational scale with standardized metadata, SEMASIA provides a reproducible foundation for studying latent geometry, evaluating alignment methods, and developing next-generation heterogeneous and interoperable AI systems.
BibTeX
@misc{pandolfo2026semasia,
  author = {Mario Edoardo Pandolfo and Enrico Grimaldi and Lorenzo Marinucci and Leonardo Di Nino and Simone Fiorellino and Sergio Barbarossa and Paolo Di Lorenzo},
  title = {{SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations}},
  year = {2026},
  eprint = {2605.09485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.09485},
}
Figure