Manifold Metric: A Loss Landscape Approach for Predicting Model Performance

Pranshu Malviya, Jerry Huang, Aristide Baratin, Quentin Fournier, Sarath ChandararXiv 2024 · arXiv

TL;DR

Researchers have developed a "Manifold Metric" that predicts the performance gains of expanding pre-trained neural networks by analyzing the geometry of their loss landscapes. By measuring the size of the manifold of linearly connected minima, this zero-cost proxy reliably ranks model expansion strategies without requiring expensive additional training, outperforming traditional baselines across both CNNs and Transformers.

What Problem Does This Paper Address?

As deep learning models and datasets grow, training from scratch becomes prohibitively expensive. A popular alternative is model expansion—taking a smaller, pre-trained model and increasing its width or depth to boost capacity. However, we currently lack a principled way to predict which expansion strategy (e.g., adding 100 channels to layer one vs. 50 channels to all layers) will yield the best final performance without actually running the costly training process.

Existing "zero-cost proxies"—metrics that estimate performance using only a single batch of data—often fail in this context. They are either computationally expensive, require some amount of training to become reliable, or simply don't correlate well with the unique dynamics of model expansion. This paper seeks a robust, geometry-driven metric to bridge this gap.

Methods

The authors introduce the Manifold Metric ($M$), which is rooted in the concept of Linear Mode Connectivity (LMC). LMC suggests that minima found by different training runs can often be connected by a straight line in parameter space without encountering a high "loss barrier."

The core idea is that a "better" model architecture will have a larger manifold of these connected, low-loss solutions. To estimate the size of this manifold, the authors propose a graph-based algorithm:

  1. Permutation: Take a pre-trained model and generate $n$ random permutations of its neurons that preserve its mathematical function.
  2. Connectivity Check: For each permutation, calculate the "loss barrier" along the linear path between the original model and the permuted version.
  3. Thresholding: If the barrier is below a threshold $\lambda$, the two are considered "connected."
  4. Metric Calculation: The metric $M$ is the ratio of connected permutations. A higher ratio suggests a wider, flatter, and more stable region in the loss landscape, which the authors link to better generalization.

To ensure a fair comparison during expansion, they use Net2Net, a technique that initializes new parameters such that the expanded model starts with the exact same output function as the smaller base model.

Datasets

The researchers evaluated their metric across several standard benchmarks:

  • Image Classification: CNNs and ResNet18 trained on CIFAR-10 and CIFAR-100.
  • Language Modeling: TinyBERT and MiniBERT models trained on WikiText-103 using a masked language modeling task.
  • Online Learning: A sequence of tasks involving MNIST, FashionMNIST, SVHN, CIFAR-10, and STL-10.

Key Insights and Ideas

  • Connectivity as a Proxy for Potential: The paper visually demonstrates that as a network is expanded, the connectivity between its minima increases, and the loss landscape appears to become "smoother" (see Figure 1).
Figure 1: LMC between minima using a MLP with one hidden layer (20 units) pre-trained on CIFAR10. The initial model (first) is expanded by increasing the hidden layer size in a function-preserving way and trained further until convergence. Final models after hidden size expansions of 2× (second) and 3× (third) show a reduced loss and enhanced local connectivity between minima.
  • Global vs. Local Geometry: Unlike metrics like GradNorm or Jacobian covariance, which look at local derivatives, the Manifold Metric captures more global structural information by spanning multiple minima.
  • Transformer Dynamics: Interestingly, the authors found that in Transformers, mode connectivity actually decreases as the model is trained after expansion. This suggests that Transformers may have distinct optimization characteristics compared to CNNs, where connectivity is generally more stable.
  • Zero-Cost Efficiency: Because the metric only requires forward passes (no backpropagation) to check connectivity, it is often faster than other proxies like SynFlow or Grasp, especially as model complexity increases.

Key Results

The Manifold Metric consistently outperformed existing zero-cost proxies in ranking candidate models. As shown in Figure 3, it was the only metric to maintain a consistently positive Kendall Tau rank correlation across all tested scenarios (CNNs, ResNets, and BERT variants).

Comparison of the Manifold Metric against other zero-cost proxies.

Quantitatively, in online learning experiments (Table 6), using the Manifold Metric to select expansion strategies led to higher final task accuracy (e.g., 76.8% on CIFAR-10) compared to other selection methods like SNIP (73.1%) or Jacov (74.5%). Furthermore, the metric proved to be computationally efficient; for a ResNet18, it was roughly 2x faster to compute than the second-best baseline.

Limitations and Future Work

While promising, the authors note several areas for further exploration:

  • Scale: The experiments focused on relatively small-scale models (CNNs and TinyBERT). Whether these correlations hold for massive Large Language Models (LLMs) remains to be seen.
  • Causality: The paper establishes a strong correlation between manifold size and performance, but a rigorous causal link between connectivity and generalization is still an open theoretical question.
  • Diminishing Returns: For very large base models that already perform near-optimally, the metric might show weaker correlations because the performance gains from expansion are naturally smaller and noisier.

The authors suggest that this geometry-driven approach could eventually be a cornerstone for more reliable and sustainable "lifelong learning" systems that grow their capacity as they encounter new data.

Summary updated

Abstract

Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model.
BibTeX
@misc{malviya2024manifold,
  author = {Pranshu Malviya and Jerry Huang and Aristide Baratin and Quentin Fournier and Sarath Chandar},
  title = {{Manifold Metric: A Loss Landscape Approach for Predicting Model Performance}},
  year = {2024},
  eprint = {2405.15895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.15895},
}
Figure