A Systematic Analysis of a Small Optimal Subspace for Physics-Informed Neural Networks: $n$-DAS $\subset \mathbb{R}^p$

Ahmad AliAhmad, Khemraj ShuklaarXiv 2026 · arXiv

TL;DR

Training a neural network often feels like searching for a needle in a high-dimensional haystack, but this paper shows that the "needle" actually lives in a tiny, predictable subspace. By selecting Hessian eigendirections based on their alignment with the gradient—rather than just their curvature magnitude—the authors introduce the Dominant Aligned Subspace (DAS), demonstrating that Physics-Informed Neural Networks (PINNs) can be trained effectively using less than 1% of their total parameter directions.

What Problem Does This Paper Address?

Modern neural networks operate in massive parameter spaces, yet empirical evidence suggests their training dynamics are often confined to much lower-dimensional manifolds. A common strategy to exploit this is to look at the Hessian (the matrix of second derivatives), specifically the directions of highest curvature (dominant eigenvalues).

However, the authors argue that this "curvature-first" approach is structurally flawed. In complex landscapes—like those of PINNs, which are notoriously ill-conditioned and anisotropic—the directions where the loss is sharpest are not necessarily the directions where the gradient is actually moving. This leads to a phenomenon called "spurious alignment," where optimizers get stuck oscillating against the "banks" of the loss landscape rather than flowing "downstream."

Figure 1: The edge of stability, as a river (an illustration of the idea). The loss is a descending river: the many flat directions run downstream (gentle slope — the channel darkens as the loss falls), the sharp direction λmax is across the channel, and the banks steepen as training progresses (denser hachures). A small learning rate settles to the centre and glides. A large one glides too, until the banks are steep enough that ηλ ≈2: from there every step overshoots the centre and lands on the opposite bank, so the trajectory stops sinking and rides the fringe — slaloming bank-to-bank along the dashed rails at the largest swing the walls can hold, while still drifting downstream. Purely schematic.

Methods

The paper introduces the Dominant Aligned Subspace (DAS). Unlike traditional methods that pick the top-$k$ eigenvectors of the Hessian, DAS ranks eigenvectors by gradient-energy scores—essentially, how much of the current gradient's "power" resides in that specific direction.

Key components of their approach include:

  • Energy Threshold ($\tau$): The subspace is defined as the smallest set of eigenvectors required to capture a specific fraction (e.g., 99% or 99.9%) of the gradient energy.
  • $n$-DAS Optimizers: They developed DAS-restricted versions of first-order (Gradient Descent) and second-order (Gauss-Newton) methods.
  • Globalization: They emphasize that line searches (like Wolfe or Armijo backtracking) are critical. These prevent the optimizer from overshooting and falling into the "spurious alignment" trap, ensuring the trajectory stays within the meaningful part of the subspace.

Datasets

The authors chose Physics-Informed Neural Networks (PINNs) as their primary testbed because PINNs induce some of the most difficult, ill-conditioned loss landscapes in deep learning. They evaluated their methods on four classic partial differential equation (PDE) benchmarks:

  1. Poisson-2D: An elliptic problem.
  2. Heat-1D: A parabolic problem.
  3. Wave-1D: A hyperbolic problem.
  4. Viscous Burgers-1D: A nonlinear problem featuring shocks.

Key Insights and Ideas

  • Alignment > Magnitude: The most important directions for optimization are those aligned with the gradient, not just those with the highest curvature. The gradient often ignores the "sharpest" directions if they don't lead toward a minimum.
  • Spectral Filtering: The DAS acts as a natural spectral filter. By discarding directions that carry little descent signal, it can actually improve solution accuracy by ignoring poorly conditioned or noisy components of the Hessian.
  • Subspace Dynamics: The DAS is not a fixed subspace; it is recomputed at every iteration. The authors found that the required dimensionality fluctuates: it expands when the gradient energy is spread out and contracts when the descent direction becomes clear.
  • Stability via Line Search: The "Edge of Stability" (EoS) is a major hurdle. The paper shows that combining DAS with a proper line search allows the optimizer to navigate these sharp landscapes without the typical loss spikes associated with fixed-step methods.

Key Results

  • Extreme Compression: On the Poisson-2D benchmark, DAS-restricted Gauss-Newton matched the accuracy of full-space training using a median of only 10–16 directions out of 1,981 parameters—less than 1% of the total space.
  • Accuracy Parity: In the Burgers PDE (a harder nonlinear case), the DAS-restricted runs were just as accurate as full-space runs, despite leaving 97% of parameter directions untouched at every step.
  • Theoretical Efficiency: For second-order methods (Gauss-Newton), the linear solve within the DAS is theoretically $2 \times 10^4$ to $3 \times 10^5$ times cheaper per iteration than a full-space solve.
  • Conditioning: Restricting updates to the DAS consistently resulted in a better-conditioned optimization problem compared to the full parameter space.

Limitations and Future Work

  • Scalability: The current implementation relies on full eigendecomposition of the Hessian, which is computationally prohibitive for large-scale models (e.g., LLMs).
  • Matrix-Free Approximations: A major direction for future work is replacing explicit Hessian formation with matrix-free methods like Krylov subspace approximations to find the DAS more efficiently.
  • Generalization: While the results are promising for PINNs, it remains to be seen if DAS provides similar benefits for general supervised learning tasks or different network architectures (like Transformers).
  • Adaptive Thresholds: The authors suggest that a fixed energy threshold ($\tau$) might not be optimal; future versions could use adaptive thresholds based on local landscape geometry.

Summary updated

Abstract

The search for a low-dimensional subspace that governs neural network training has been a persistent goal in optimization. Yet subspaces defined by Hessian eigenvalue magnitude, such as the dominant/bulk decomposition underlying Dom-SGD and Bulk-SGD, have repeatedly failed to capture where optimization takes place, and adaptive Krylov-based alternatives built on energy-fraction thresholds show promise but still struggle at lower eigenmodes. We argue this failure is structural: the directions of highest curvature are not, in general, the directions along which training moves. We introduce the dominant aligned subspace (DAS), constructed from a full eigendecomposition of the Hessian but selected by gradient alignment rather than eigenvalue magnitude, as a tool for understanding optimization dynamics and a foundation for designing methods that exploit them. To expose this structure under the hardest conditions, we use physics-informed neural networks (PINNs) as a testbed: differential operators from the parabolic, elliptic, and hyperbolic classes induce severely ill-conditioned, anisotropic loss landscapes. Across Poisson, heat, wave, and viscous Burgers benchmarks DAS-restricted training with a line search matches full-space training even where the projected Hessian remains ill-conditioned, showing empirically that convergence does not require all Hessian eigendirections but only a low-dimensional, gradient-aligned subset. We extend the analysis to Gauss-Newton within the DAS and discuss what this structure implies for future subspace-aware optimizers.
BibTeX
@misc{aliahmad2026systematic,
  author = {Ahmad AliAhmad and Khemraj Shukla},
  title = {{A Systematic Analysis of a Small Optimal Subspace for Physics-Informed Neural Networks: $n$-DAS $\textbackslash\{\}subset \textbackslash\{\}mathbb\{R\}^p$}},
  year = {2026},
  eprint = {2611.23773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23773},
}
Figure