Roughness Index for Loss Landscapes of Neural Network Models of Partial Differential Equations

Keke Wu, Xiangru Jian, Rui Du, Jingrun Chen, Xiang Zhou

TL;DR

This paper introduces a novel "roughness index" (RI) to quantify the complexity of loss landscapes for neural networks solving partial differential equations (PDEs). They demonstrate that the Deep Galerkin Method (DGM) generally exhibits less rough loss landscapes than the Deep Ritz Method (DRM), potentially explaining DGM's superior performance in practice.

What Problem Does This Paper Address?

Training neural networks to solve PDEs involves minimizing a complex, high-dimensional loss function. Understanding the "shape" or "landscape" of this loss function is crucial for effective training and generalization. Traditional methods for analyzing these landscapes, like examining Hessian matrices, are often limited to local quadratic approximations and struggle with the inherent non-convexity and high dimensionality. This paper aims to provide a more robust and generalizable quantitative measure of loss landscape roughness, specifically comparing two popular neural network-based PDE solvers: the Deep Galerkin Method (DGM) and the Deep Ritz Method (DRM).

Methods

The core contribution is the development of a Roughness Index (RI). This index is derived from the normalized total variation (TV) of 1D projections of the loss function onto randomly sampled directions.

Here's a breakdown of the method:

  • 1D Projections: The high-dimensional loss function is projected onto random 1D lines. This is achieved by taking a reference point (often a local minimizer) and moving along random directions d.
  • Filter-wise Normalization: To make these projections comparable across different directions and scales, a filter-wise normalization is applied.
  • Normalized Total Variation (TV): For each 1D projection, the normalized total variation is computed. TV measures the "wiggliness" or oscillation of a function. The normalization accounts for the range and domain of the projected function.
  • Roughness Index (RI): The RI is then calculated as the ratio of the standard deviation (σ) to the mean (μ) of these normalized TV values across multiple random directions. A higher RI indicates a more anisotropic and rough landscape.

This approach is novel because it:

  • Moves beyond local quadratic approximations (like Hessian analysis).
  • Is applicable to both convex and non-convex landscapes.
  • Quantifies landscape roughness in a way that is sensitive to oscillations and barriers.

Datasets

The paper uses synthetic datasets generated by solving various PDEs:

  • 1D Poisson Equation: Used extensively for initial analysis and visualization.
  • 3D Equation with Low-Regularity Solution: Explores the method's applicability to more challenging scenarios.
  • High-Dimensional Poisson Equation (d=10): Tests scalability.
  • 1D Wave Equation: Further validation of the method.

The "datasets" here are not traditional data samples but rather the numerical solutions and corresponding loss functions generated by these PDE problems.

Key Insights and Ideas

  • Loss Landscape Roughness Matters: The paper strongly suggests that the "roughness" of a loss landscape significantly impacts the training dynamics and the quality of the obtained solution for PDEs.
  • DGM vs. DRM: A Roughness Difference: A consistent finding is that the DGM generally produces significantly less rough loss landscapes around its local minimizers compared to the DRM.
  • Anisotropy as a Source of Roughness: The higher roughness in DRM's landscape is attributed to its greater anisotropy (higher standard deviation of directional TV), rather than just a higher average TV.
  • Gradient Descent Trajectory: The RI exhibits an "increasing-then-decreasing" pattern along the gradient descent path, suggesting a transition from a smooth region to a rougher one before settling into a flatter minimum. This "flat-rough-flat" transition is a key observation.
  • RI as a Predictive Tool: The RI appears to be a good indicator of training difficulty and potential solution quality, correlating with observed differences in accuracy between DGM and DRM.

Key Results

  • Quantitative Comparison: In all tested scenarios (1D, 3D, 10D Poisson, 1D wave equation), the DGM consistently showed a lower roughness index than the DRM at local minimizers. For example, in Table IV, for ResNet with M=50, the RI for DGM was 0.0455 while for DRM it was 0.2472.
  • Visualization Confirmation: Contour plots of the loss landscapes around local minimizers visually confirmed the rougher, more oscillatory nature of the DRM compared to the DGM.
  • Training Dynamics: Figure 5 shows that while both DGM and DRM start in smooth regions, the DRM's RI increases significantly as it approaches a minimizer, whereas DGM's RI remains relatively low.
  • Accuracy Correlation: The paper notes that DGM generally achieves better accuracy in solving PDEs, which they hypothesize is linked to its less rough loss landscape.

Limitations and Future Work

  • Mathematical Rigor: The authors acknowledge that a rigorous mathematical connection between the roughness index and the observed performance differences (accuracy, training difficulty) is still lacking, despite strong empirical evidence.
  • Generalizability: While the method is proposed as general, its extensive validation is limited to PDE-solving neural networks. Further investigation into other machine learning tasks (e.g., image classification) is suggested.
  • Parameter Sensitivity: The choice of interval length l for the 1D projections is noted as important, and testing a range of values is recommended. The paper finds l=0.01 to be representative for their examples.
  • Exploration of All Minima: The RI is computed at specific local minimizers. The paper acknowledges that it's not feasible to explore all possible local minima to confirm the findings universally.
  • Future Work: The authors suggest applying the RI to other machine learning tasks and conducting rigorous mathematical analysis to establish a stronger theoretical foundation. They also propose further investigation into the "flat-rough-flat" transition observed during gradient descent.

Summary updated

Abstract

Loss landscape is a useful tool for characterizing and comparing neural network models. The main challenge for analysis of loss landscape for the deep neural networks is that they are generally highly nonconvex in very high-dimensional space. In this paper, we develop the “roughness” concept for understanding such landscapes in high dimensions and apply this technique to study two neural network models arising from solving differential equations. Our main innovation is the proposal of a well-defined and easy-to-compute roughness index (RI) which is based on the mean and variance of the (normalized) total variation for one-dimensional functions projected on randomly sampled directions. A large RI at the local minimizer indicates an oscillatory landscape profile and indicates a severe challenge for the first-order optimization method. Particularly, we observe the increasing-then-decreasing pattern for RI along the gradient descent path in most models. We apply our method to two types of loss functions used to solve partial differential equations (PDEs) when the solution of PDE is parametrized by neural networks. Our empirical results on these PDE problems reveal important and consistent observations that the landscapes from the deep Galerkin method around its local minimizers are less rough than the deep Ritz method.
BibTeX
@inproceedings{wu2023roughness,
  author = {Keke Wu and Xiangru Jian and Rui Du and Jingrun Chen and Xiang Zhou},
  title = {{Roughness Index for Loss Landscapes of Neural Network Models of Partial Differential Equations}},
  booktitle = {IEEE Big Data},
  year = {2023},
  doi = {10.1109/bigdata59044.2023.10386909},
}
Figure