Abstract
Understanding the high-dimensional loss landscapes of neural networks is critical for diagnosing training failures and improving generalization. However, traditional visualization techniques often provide a distorted view of the optimization process. This paper presents Neuro-Visualizer, a novel tool that utilizes a neural auto-encoder to create high-fidelity, non-linear 2D manifolds of parameter space. Unlike linear methods such as Principal Component Analysis (PCA), Neuro-Visualizer can "wrap" around complex training trajectories, preserving the "curvy" nature of optimization paths. Furthermore, its decoder-based architecture enables an inverse mapping, allowing for the calculation of actual loss values across the entire visualization. We demonstrate its effectiveness on complex physics-guided neural networks, where it reveals intricate local minima structures invisible to standard techniques.
Introduction
The "loss landscape"—the high-dimensional surface representing a model's error relative to its parameters—is a fundamental concept in deep learning. Visualizing this space is essential for understanding why certain models converge while others fail. However, projecting millions of dimensions onto a 2D screen is inherently lossy.
Standard techniques like PCA take linear "slices" through the landscape, which are often inaccurate far from the origin and fail to capture non-linear training dynamics. Conversely, non-linear methods like t-SNE or UMAP are effective for clustering but lack an inverse transform, preventing researchers from mapping 2D points back to actual model parameters to calculate loss. Neuro-Visualizer addresses these gaps by providing a programmable, non-linear, and invertible visualization framework.
Methodology
Neuro-Visualizer employs a neural auto-encoder to learn a mapping between the high-dimensional parameter space and a 2D grid. The framework introduces a constrained optimization approach that allows users to tailor the visualization to their specific needs:
- Location Anchoring: Users can "pin" specific models (e.g., the initial and final states) to fixed grid coordinates, ensuring the trajectory is well-distributed across the view.
- Adaptive Grid Scaling ($L_{grid}$): The tool can increase grid density near the training trajectory, providing high-resolution detail where the model actually traveled while "zooming out" on less relevant regions.
- Inverse Mapping: The use of a decoder ensures that every point on the 2D plot corresponds to a set of weights, enabling the plotting of actual loss values across the manifold.
Experimental Setup
The method was evaluated using Knowledge-Guided Machine Learning (KGML) applications, characterized by complex, non-convex loss landscapes due to competing data and physics constraints:
- CoPhy-PGNN: A physics-guided network for solving the Schrödinger equation.
- PINNs (Physics-Informed Neural Networks): Applied to Convection Partial Differential Equations (PDEs).
Key Insights
A primary finding of this work is that linearity is deceptive. Linear projections like PCA often suggest smooth paths or artificial fluctuations that do not exist in the true high-dimensional space. Neuro-Visualizer reveals that these "smooth" paths are often actually "basins" of complex local minima.
The tool also allows for the visual verification of theoretical hypotheses, such as the Neural Tangent Kernel (NTK) failure modes in PINNs. By visualizing how different loss terms (e.g., boundary conditions vs. physics residuals) converge, researchers can identify "pathologies" in the optimization process that are invisible in standard 1D loss plots.
Results
Quantitative evaluations demonstrate that Neuro-Visualizer significantly outperforms traditional methods in projection accuracy and loss fidelity. In the CoPhy-PGNN task, Neuro-Visualizer achieved an average relative physics loss error of 0.0095, compared to 1.6782 for PCA. Similarly, its projection error was nearly three orders of magnitude lower than PCA (0.0005 vs. 0.2832).
Qualitatively, the tool provides a visual explanation for the success of various optimization strategies. For instance, loss balancing techniques like GradNorm were shown to lead models into significantly different "basins," illustrating why certain hyperparameters yield better convergence.
Conclusion and Future Work
Neuro-Visualizer offers a powerful, "programmable" lens for exploring neural network optimization. While currently limited by the scalability of the auto-encoder to models with millions of parameters, it provides a blueprint for high-fidelity visualization. Future work will focus on developing more efficient encoding schemes for massive architectures and using the tool to study the relationship between landscape sharpness and generalization.
Summary updated