A Model Zoo on Phase Transitions in Neural Networks

Konstantin Schürholt, Léo Meynent, Yefan Zhou, Haiquan Lu, Yaoqing Yang, Damian Borth

TL;DR

This paper introduces 12 large-scale "Phase Model Zoos" containing over 2,500 unique neural networks and 60,000 checkpoints across vision, language, and scientific computing. By leveraging concepts from statistical physics, the authors systematically map out the "loss landscape phases" of these models, providing a structured dataset to advance Weight Space Learning (WSL) and phase-aware deep learning.

What Problem Does This Paper Address?

As deep learning scales, researchers are increasingly treating the weights of trained neural networks as a new form of data—a field known as Weight Space Learning (WSL). WSL can be used to predict model generalization, detect poisoned models, or even generate new weights directly. However, developing WSL methods requires large populations of trained models ("model zoos").

Currently, existing model zoos are unstructured and lack a principled definition of diversity. At the same time, insights from statistical physics have shown that neural networks undergo distinct phase transitions during training. Models within the same phase share homogeneous properties, while models in different phases behave qualitatively differently. This paper bridges the gap by creating the first model zoos systematically structured around these loss landscape phases to provide a quantifiable metric of model diversity.

Methods

The authors systematically vary two main axes derived from statistical physics to navigate the loss landscape:

  1. Relative Load: The quantity or quality of data relative to model capacity (varied by changing model width or problem complexity).
  2. Temperature: The noisiness of the training process (varied by adjusting batch size or learning rate).
Figure 1: Overview of the approach. Left: Phases of Neural Networks with qualitatively different properties have been identified (Martin and Mahoney, 2019a; Yang et al., 2021). Middle: We propose to use phases as a quantifiable diversity metric and generate datasets of trained models by varying their datasets D, architectures A and training parameters λ s.t. they cover all known phases across different domains. Right: Phases manifest beyond evaluation metrics in applications of pre-trained models, e.g., in weight averaging. Our Phase Model Zoos allow systematic evaluation of methods that use pre-trained models.

To categorize the models into specific phases, the authors compute several key loss landscape metrics:

  • Hessian-based metrics (Top Eigenvalue and Trace): To measure the local sharpness of minima.
  • Mode Connectivity: To evaluate the presence of low-loss paths between different local minima.
  • Centered Kernel Alignment (CKA) Similarity: To measure representation similarity across different random seeds.

Using these metrics, models are mapped into a five-phase taxonomy: Phase I (underfitted & undersized), Phase II (underfitted), Phase III (undersized), Phase IV-A (overfitted), and Phase IV-B (generalizing).

Five-phase taxonomy in NN hyperparameter space

Datasets

The authors introduce 12 distinct model zoos spanning three major domains:

  • Computer Vision (CV): ResNet-18, ResNet-50, and Vision Transformers (ViT) trained on SVHN, CIFAR-10, CIFAR-100, and TinyImageNet.
  • Natural Language Processing (NLP): GPT-2 models trained on OpenWebText.
  • Scientific ML (SciML): Physics-Informed Neural Networks (PINNs) trained on a simulated 1D Convection partial differential equation (PDE).

Key Insights and Ideas

  • Phases Generalize Universally: One of the most striking findings is that these five training phases consistently emerge across completely different architectures (CNNs, Transformers, MLPs) and vastly different tasks (image classification, language generation, and physical simulations).
  • Downstream Tasks are Phase-Dependent: A model's phase heavily dictates how well it will perform in post-training applications. For example, Phase IV (well-trained) networks preserve accuracy much better under heavy pruning, while weight-averaging techniques like Git Re-Basin fail when applied to models from negative-connectivity phases (Phases I or III).
  • Weights Predict the Landscape: Simple linear probes can partially predict complex, expensive global loss landscape metrics (like the log Hessian trace or CKA similarity) using nothing but raw weight statistics (means and quintiles per layer).
Phase transitions in downstream methods

Key Results

  • The Dataset: The final release consists of ~2.5k unique models and over 60k checkpoints, trained using substantial compute (30 days on 16 DGX H100 GPUs).
  • Phase-Aware Hyperparameter Optimization: The authors designed a simple decision rule using loss landscape metrics to guide hyperparameter adjustments in a single step. This phase-aware optimization significantly outperformed a standard random search baseline across multiple datasets (e.g., achieving a performance gain of $0.116 \pm 0.007$ vs. $0.037 \pm 0.012$ for random search on TinyImageNet).
  • Weight-Space Property Prediction: Linear probes achieved high $R^2$ scores when predicting test accuracy directly from raw weight statistics (e.g., $R^2 = 0.92$ on SVHN and $R^2 = 0.90$ on TinyImageNet).

Limitations and Future Work

While the dataset successfully demonstrates phase transitions across multiple domains, the majority of the model zoos are still concentrated on classification tasks within computer vision. The authors note that extending this exhaustive phase mapping to a wider variety of tasks, larger model scales, and more diverse scientific domains remains an important direction for future work.

Summary updated

Abstract

Using the weights of trained Neural Network (NN) models as data modality has recently gained traction as a research field - dubbed Weight Space Learning (WSL). Multiple recent works propose WSL methods to analyze models, evaluate methods, or synthesize weights. Weight space learning methods require populations of trained models as datasets for development and evaluation. However, existing collections of models - called `model zoos' - are unstructured or follow a rudimentary definition of diversity. In parallel, work rooted in statistical physics has identified phases and phase transitions in NN models. Models are homogeneous within the same phase but qualitatively differ from one phase to another. We combine the idea of `model zoos' with phase information to create a controlled notion of diversity in populations. We introduce 12 large-scale zoos that systematically cover known phases and vary over model architecture, size, and datasets. These datasets cover different modalities, such as computer vision, natural language processing, and scientific ML. For every model, we compute loss landscape metrics and validate full coverage of the phases. With this dataset, we provide the community with a resource with a wide range of potential applications for WSL and beyond. Evidence suggests the loss landscape phase plays a role in applications such as model training, analysis, or sparsification. We demonstrate this in an exploratory study of the downstream methods like transfer learning or model weights averaging.
BibTeX
@inproceedings{schurholt2025model,
  author = {Konstantin Schürholt and Léo Meynent and Yefan Zhou and Haiquan Lu and Yaoqing Yang and Damian Borth},
  title = {{A Model Zoo on Phase Transitions in Neural Networks}},
  booktitle = {Accepted by DMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=zJRWvNpdIr},
}
Figure