Roughness Index for Loss Landscapes of Neural Network Models of Partial Differential Equations

Keke Wu, Xiangru Jian, Rui Du, Jingrun Chen, Xiang Zhou

TL;DR

This paper introduces the Roughness Index (RI), a novel metric based on total variation that quantifies the multi-scale oscillations of high-dimensional neural network loss landscapes. By applying RI to physics-informed neural networks, the authors demonstrate that the Deep Galerkin Method (DGM) creates significantly smoother, less oscillatory landscapes around local minima than the Deep Ritz Method (DRM), explaining DGM's superior numerical accuracy.

What Problem Does This Paper Address?

When training deep neural networks to solve partial differential equations (PDEs), the choice of loss function drastically alters the optimization landscape. Visualizing or characterizing these high-dimensional, non-convex landscapes is notoriously difficult.

Existing metrics, like Hessian eigenvalues, only measure local "flatness" under the assumption that the landscape is locally smooth and quadratic. They completely miss small-scale, highly oscillatory "wells"—or roughness—that can trap first-order optimizers like Adam or SGD. The authors address this gap by defining a quantitative measure of landscape roughness that goes beyond infinitesimal quadratic approximations.

Two 1D functions demonstrating how Total Variation captures roughness.

Methods

The authors propose the Roughness Index (RI). Instead of relying on derivatives at a single point, RI uses 1D random projections. For a given parameter point $\theta$, the loss function is sliced along $M$ randomly sampled Gaussian directions using filter-wise normalization.

For each direction, they compute a modified Total Variation (TV) over a neighborhood $[-l, l]$. Crucially, this modified TV normalizes both the domain size and the function's range to fit a unit square, making it invariant to simple scaling or dilation. The RI is then defined as the ratio of the standard deviation ($\sigma$) to the mean ($\mu$) of these normalized TVs across all random directions:

$$I(J; \theta) = \frac{\text{std}_d(T(f_d))}{\mathbb{E}_d(T(f_d))}$$

A higher RI indicates that the landscape's roughness is highly anisotropic (varying wildly depending on the direction). The authors use RI to compare two prominent PDE-solving frameworks:

  1. Deep Galerkin Method (DGM): Minimizes the $L^2$ residual of the PDE operator.
  2. Deep Ritz Method (DRM): Minimizes a variational energy functional.

Datasets

Rather than standard computer vision datasets, this paper evaluates its method on synthetic PDE benchmarks common in scientific computing:

  • 1D Poisson Equation: A smooth, well-behaved baseline.
  • 3D Poisson Equation: Features a low-regularity solution that is continuous but non-differentiable at the origin.
  • 10D Poisson Equation: Tests the scalability of the index in high-dimensional parameter spaces.
  • 1D Wave Equation: A time-dependent hyperbolic PDE where DRM is not applicable.

Key Insights and Ideas

  • DGM vs. DRM Roughness: Around local minimizers, DGM consistently exhibits a significantly lower RI than DRM. This implies DGM's landscape is smoother and less oscillatory, which aligns with empirical observations that DGM achieves better accuracy.
  • Anisotropy, Not Mean Roughness: Surprisingly, the average roughness ($\mu$) across directions is nearly identical for both DGM and DRM. The difference in RI is driven entirely by the standard deviation ($\sigma$). DRM's landscape is highly anisotropic—extremely rough in some directions and smooth in others—making optimization more treacherous.
  • The "Flat-Rough-Flat" Transition: By tracking RI along the gradient descent path, the authors discovered an increasing-then-decreasing pattern. At initialization, the landscape is globally smooth and monotonic (low RI). As the optimizer approaches the valley, it enters a highly oscillatory, rough region before finally settling into a smooth local minimum basin.
The Roughness Index along the gradient descent path, showing an increasing-then-decreasing pattern.

Key Results

  • Quantifying the Gap: The RI for DGM was consistently an order of magnitude smaller than DRM across various network widths and architectures (e.g., $I_{DGM} \approx 0.028$ vs $I_{DRM} \approx 0.674$ for a standard ResNet setup).
  • Visual Validation: 2D contour visualizations qualitatively confirm the RI metrics: DGM shows wide, smooth basins, while DRM displays highly jagged, oscillatory contours around its minima.
2D loss landscape visualizations showing smoother basins for DGM (left) vs. oscillatory contours for DRM (right).

Limitations and Future Work

  • Lack of Theoretical Proof: While RI correlates strongly with optimization performance, a rigorous mathematical proof linking a lower RI to better generalization or convergence speed is still missing.
  • Scope of Application: The study focuses exclusively on neural networks solving PDEs. Whether the "flat-rough-flat" optimization trajectory or the specific anisotropy findings hold true for standard machine learning tasks like image classification or language modeling remains an open question for future empirical validation.

Summary updated

Abstract

Loss landscape is a useful tool to characterize and compare neural network models. The main challenge for analysis of loss landscape for the deep neural networks is that they are generally highly non-convex in very high dimensional space. In this paper, we develop "the roughness"concept for understanding such landscapes in high dimensions and apply this technique to study two neural network models arising from solving differential equations. Our main innovation is the proposal of a well-defined and easy-to-compute roughness index (RI) which is based on the mean and variance of the (normalized) total variation for one-dimensional functions projected on randomly sampled directions. A large RI at the local minimizer hints an oscillatory landscape profile and indicates a severe challenge for the first-order optimization method. Particularly, we observe the increasing-then-decreasing pattern for RI along the gradient descent path in most models. We apply our method to two types of loss functions used to solve partial differential equations (PDEs) when the solution of PDE is parametrized by neural networks. Our empirical results on these PDE problems reveal important and consistent observations that the landscapes from the deep Galerkin method around its local minimizers are less rough than the deep Ritz method.
BibTeX
@inproceedings{wu2023roughness,
  author = {Keke Wu and Xiangru Jian and Rui Du and Jingrun Chen and Xiang Zhou},
  title = {{Roughness Index for Loss Landscapes of Neural Network Models of Partial Differential Equations}},
  booktitle = {IEEE Big Data},
  year = {2023},
  doi = {10.1109/bigdata59044.2023.10386909},
}
Figure