Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Method

Mohannad Elhamod, Anuj KarpatnearXiv 2023 · DOI · arXiv

Abstract

Understanding the high-dimensional loss landscapes of neural networks is critical for diagnosing training failures and improving generalization. However, traditional visualization techniques often provide a distorted view of the optimization process. This paper presents Neuro-Visualizer, a novel tool that utilizes a neural auto-encoder to create high-fidelity, non-linear 2D manifolds of parameter space. Unlike linear methods such as Principal Component Analysis (PCA), Neuro-Visualizer can "wrap" around complex training trajectories, preserving the "curvy" nature of optimization paths. Furthermore, its decoder-based architecture enables an inverse mapping, allowing for the calculation of actual loss values across the entire visualization. We demonstrate its effectiveness on complex physics-guided neural networks, where it reveals intricate local minima structures invisible to standard techniques.

Introduction

The "loss landscape"—the high-dimensional surface representing a model's error relative to its parameters—is a fundamental concept in deep learning. Visualizing this space is essential for understanding why certain models converge while others fail. However, projecting millions of dimensions onto a 2D screen is inherently lossy.

Standard techniques like PCA take linear "slices" through the landscape, which are often inaccurate far from the origin and fail to capture non-linear training dynamics. Conversely, non-linear methods like t-SNE or UMAP are effective for clustering but lack an inverse transform, preventing researchers from mapping 2D points back to actual model parameters to calculate loss. Neuro-Visualizer addresses these gaps by providing a programmable, non-linear, and invertible visualization framework.

Methodology

Neuro-Visualizer employs a neural auto-encoder to learn a mapping between the high-dimensional parameter space and a 2D grid. The framework introduces a constrained optimization approach that allows users to tailor the visualization to their specific needs:

  1. Location Anchoring: Users can "pin" specific models (e.g., the initial and final states) to fixed grid coordinates, ensuring the trajectory is well-distributed across the view.
  2. Adaptive Grid Scaling ($L_{grid}$): The tool can increase grid density near the training trajectory, providing high-resolution detail where the model actually traveled while "zooming out" on less relevant regions.
  3. Inverse Mapping: The use of a decoder ensures that every point on the 2D plot corresponds to a set of weights, enabling the plotting of actual loss values across the manifold.

Experimental Setup

The method was evaluated using Knowledge-Guided Machine Learning (KGML) applications, characterized by complex, non-convex loss landscapes due to competing data and physics constraints:

  • CoPhy-PGNN: A physics-guided network for solving the Schrödinger equation.
  • PINNs (Physics-Informed Neural Networks): Applied to Convection Partial Differential Equations (PDEs).

Key Insights

A primary finding of this work is that linearity is deceptive. Linear projections like PCA often suggest smooth paths or artificial fluctuations that do not exist in the true high-dimensional space. Neuro-Visualizer reveals that these "smooth" paths are often actually "basins" of complex local minima.

Neuro-Visualizer (zoomed-in) reveals the intricate structure of local minima that linear methods often smooth over.

The tool also allows for the visual verification of theoretical hypotheses, such as the Neural Tangent Kernel (NTK) failure modes in PINNs. By visualizing how different loss terms (e.g., boundary conditions vs. physics residuals) converge, researchers can identify "pathologies" in the optimization process that are invisible in standard 1D loss plots.

Results

Quantitative evaluations demonstrate that Neuro-Visualizer significantly outperforms traditional methods in projection accuracy and loss fidelity. In the CoPhy-PGNN task, Neuro-Visualizer achieved an average relative physics loss error of 0.0095, compared to 1.6782 for PCA. Similarly, its projection error was nearly three orders of magnitude lower than PCA (0.0005 vs. 0.2832).

Comparison of loss landscape visualization methods across different techniques: (a) PCA, (b) PCA zoomed, (c) Kernel-PCA, (d) Neuro-Visualizer, (e) Neuro-Visualizer zoomed, and (f) UMAP.

Qualitatively, the tool provides a visual explanation for the success of various optimization strategies. For instance, loss balancing techniques like GradNorm were shown to lead models into significantly different "basins," illustrating why certain hyperparameters yield better convergence.

Conclusion and Future Work

Neuro-Visualizer offers a powerful, "programmable" lens for exploring neural network optimization. While currently limited by the scalability of the auto-encoder to models with millions of parameters, it provides a blueprint for high-fidelity visualization. Future work will focus on developing more efficient encoding schemes for massive architectures and using the tool to study the relationship between landscape sharpness and generalization.

Summary updated

Abstract

In recent years, there has been a growing interest in visualizing the loss landscape of neural networks. Linear landscape visualization methods, such as principal component analysis, have become widely used as they intuitively help researchers study neural networks and their training process. However, these linear methods suffer from limitations and drawbacks due to their lack of flexibility and low fidelity at representing the high dimensional landscape. In this paper, we present a novel auto-encoder-based non-linear landscape visualization method called Neuro-Visualizer that addresses these shortcoming and provides useful insights about neural network loss landscapes. To demonstrate its potential, we run experiments on a variety of problems in two separate applications of knowledge-guided machine learning (KGML). Our findings show that Neuro-Visualizer outperforms other linear and non-linear baselines and helps corroborate, and sometime challenge, claims proposed by machine learning community. All code and data used in the experiments of this paper are available at an anonymous link https://anonymous.4open.science/r/NeuroVisualizer-FDD6
BibTeX
@misc{elhamod2023neuro,
  author = {Mohannad Elhamod and Anuj Karpatne},
  title = {{Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Method}},
  year = {2023},
  eprint = {2309.14601},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.14601},
}
Figure