TL;DR
This paper introduces the Roughness Index (RI), a novel metric based on total variation that quantifies the multi-scale oscillations of high-dimensional neural network loss landscapes. By applying RI to physics-informed neural networks, the authors demonstrate that the Deep Galerkin Method (DGM) creates significantly smoother, less oscillatory landscapes around local minima than the Deep Ritz Method (DRM), explaining DGM's superior numerical accuracy.
What Problem Does This Paper Address?
When training deep neural networks to solve partial differential equations (PDEs), the choice of loss function drastically alters the optimization landscape. Visualizing or characterizing these high-dimensional, non-convex landscapes is notoriously difficult.
Existing metrics, like Hessian eigenvalues, only measure local "flatness" under the assumption that the landscape is locally smooth and quadratic. They completely miss small-scale, highly oscillatory "wells"—or roughness—that can trap first-order optimizers like Adam or SGD. The authors address this gap by defining a quantitative measure of landscape roughness that goes beyond infinitesimal quadratic approximations.
Methods
The authors propose the Roughness Index (RI). Instead of relying on derivatives at a single point, RI uses 1D random projections. For a given parameter point $\theta$, the loss function is sliced along $M$ randomly sampled Gaussian directions using filter-wise normalization.
For each direction, they compute a modified Total Variation (TV) over a neighborhood $[-l, l]$. Crucially, this modified TV normalizes both the domain size and the function's range to fit a unit square, making it invariant to simple scaling or dilation. The RI is then defined as the ratio of the standard deviation ($\sigma$) to the mean ($\mu$) of these normalized TVs across all random directions:
$$I(J; \theta) = \frac{\text{std}_d(T(f_d))}{\mathbb{E}_d(T(f_d))}$$
A higher RI indicates that the landscape's roughness is highly anisotropic (varying wildly depending on the direction). The authors use RI to compare two prominent PDE-solving frameworks:
- Deep Galerkin Method (DGM): Minimizes the $L^2$ residual of the PDE operator.
- Deep Ritz Method (DRM): Minimizes a variational energy functional.
Datasets
Rather than standard computer vision datasets, this paper evaluates its method on synthetic PDE benchmarks common in scientific computing:
- 1D Poisson Equation: A smooth, well-behaved baseline.
- 3D Poisson Equation: Features a low-regularity solution that is continuous but non-differentiable at the origin.
- 10D Poisson Equation: Tests the scalability of the index in high-dimensional parameter spaces.
- 1D Wave Equation: A time-dependent hyperbolic PDE where DRM is not applicable.
Key Insights and Ideas
- DGM vs. DRM Roughness: Around local minimizers, DGM consistently exhibits a significantly lower RI than DRM. This implies DGM's landscape is smoother and less oscillatory, which aligns with empirical observations that DGM achieves better accuracy.
- Anisotropy, Not Mean Roughness: Surprisingly, the average roughness ($\mu$) across directions is nearly identical for both DGM and DRM. The difference in RI is driven entirely by the standard deviation ($\sigma$). DRM's landscape is highly anisotropic—extremely rough in some directions and smooth in others—making optimization more treacherous.
- The "Flat-Rough-Flat" Transition: By tracking RI along the gradient descent path, the authors discovered an increasing-then-decreasing pattern. At initialization, the landscape is globally smooth and monotonic (low RI). As the optimizer approaches the valley, it enters a highly oscillatory, rough region before finally settling into a smooth local minimum basin.
Key Results
- Quantifying the Gap: The RI for DGM was consistently an order of magnitude smaller than DRM across various network widths and architectures (e.g., $I_{DGM} \approx 0.028$ vs $I_{DRM} \approx 0.674$ for a standard ResNet setup).
- Visual Validation: 2D contour visualizations qualitatively confirm the RI metrics: DGM shows wide, smooth basins, while DRM displays highly jagged, oscillatory contours around its minima.
Limitations and Future Work
- Lack of Theoretical Proof: While RI correlates strongly with optimization performance, a rigorous mathematical proof linking a lower RI to better generalization or convergence speed is still missing.
- Scope of Application: The study focuses exclusively on neural networks solving PDEs. Whether the "flat-rough-flat" optimization trajectory or the specific anisotropy findings hold true for standard machine learning tasks like image classification or language modeling remains an open question for future empirical validation.
Summary updated