TL;DR
This paper introduces 12 large-scale "Phase Model Zoos" containing over 2,500 unique neural networks and 60,000 checkpoints across vision, language, and scientific computing. By leveraging concepts from statistical physics, the authors systematically map out the "loss landscape phases" of these models, providing a structured dataset to advance Weight Space Learning (WSL) and phase-aware deep learning.
What Problem Does This Paper Address?
As deep learning scales, researchers are increasingly treating the weights of trained neural networks as a new form of data—a field known as Weight Space Learning (WSL). WSL can be used to predict model generalization, detect poisoned models, or even generate new weights directly. However, developing WSL methods requires large populations of trained models ("model zoos").
Currently, existing model zoos are unstructured and lack a principled definition of diversity. At the same time, insights from statistical physics have shown that neural networks undergo distinct phase transitions during training. Models within the same phase share homogeneous properties, while models in different phases behave qualitatively differently. This paper bridges the gap by creating the first model zoos systematically structured around these loss landscape phases to provide a quantifiable metric of model diversity.
Methods
The authors systematically vary two main axes derived from statistical physics to navigate the loss landscape:
- Relative Load: The quantity or quality of data relative to model capacity (varied by changing model width or problem complexity).
- Temperature: The noisiness of the training process (varied by adjusting batch size or learning rate).
To categorize the models into specific phases, the authors compute several key loss landscape metrics:
- Hessian-based metrics (Top Eigenvalue and Trace): To measure the local sharpness of minima.
- Mode Connectivity: To evaluate the presence of low-loss paths between different local minima.
- Centered Kernel Alignment (CKA) Similarity: To measure representation similarity across different random seeds.
Using these metrics, models are mapped into a five-phase taxonomy: Phase I (underfitted & undersized), Phase II (underfitted), Phase III (undersized), Phase IV-A (overfitted), and Phase IV-B (generalizing).
Datasets
The authors introduce 12 distinct model zoos spanning three major domains:
- Computer Vision (CV): ResNet-18, ResNet-50, and Vision Transformers (ViT) trained on SVHN, CIFAR-10, CIFAR-100, and TinyImageNet.
- Natural Language Processing (NLP): GPT-2 models trained on OpenWebText.
- Scientific ML (SciML): Physics-Informed Neural Networks (PINNs) trained on a simulated 1D Convection partial differential equation (PDE).
Key Insights and Ideas
- Phases Generalize Universally: One of the most striking findings is that these five training phases consistently emerge across completely different architectures (CNNs, Transformers, MLPs) and vastly different tasks (image classification, language generation, and physical simulations).
- Downstream Tasks are Phase-Dependent: A model's phase heavily dictates how well it will perform in post-training applications. For example, Phase IV (well-trained) networks preserve accuracy much better under heavy pruning, while weight-averaging techniques like Git Re-Basin fail when applied to models from negative-connectivity phases (Phases I or III).
- Weights Predict the Landscape: Simple linear probes can partially predict complex, expensive global loss landscape metrics (like the log Hessian trace or CKA similarity) using nothing but raw weight statistics (means and quintiles per layer).
Key Results
- The Dataset: The final release consists of ~2.5k unique models and over 60k checkpoints, trained using substantial compute (30 days on 16 DGX H100 GPUs).
- Phase-Aware Hyperparameter Optimization: The authors designed a simple decision rule using loss landscape metrics to guide hyperparameter adjustments in a single step. This phase-aware optimization significantly outperformed a standard random search baseline across multiple datasets (e.g., achieving a performance gain of $0.116 \pm 0.007$ vs. $0.037 \pm 0.012$ for random search on TinyImageNet).
- Weight-Space Property Prediction: Linear probes achieved high $R^2$ scores when predicting test accuracy directly from raw weight statistics (e.g., $R^2 = 0.92$ on SVHN and $R^2 = 0.90$ on TinyImageNet).
Limitations and Future Work
While the dataset successfully demonstrates phase transitions across multiple domains, the majority of the model zoos are still concentrated on classification tasks within computer vision. The authors note that extending this exhaustive phase mapping to a wider variety of tasks, larger model scales, and more diverse scientific domains remains an important direction for future work.
Summary updated