Introduction to optimization methods for training SciML models

Alena Kopaničáková, Elisa Riccietti

This academic paper, "Introduction to optimization methods for training SciML models," serves as a comprehensive bridge between the worlds of classical machine learning (ML) and scientific computing. It provides a rigorous look at why the "standard" tools of AI often stumble when applied to physics and how we can fix them.

TL;DR

While classical ML thrives on stochastic, data-driven optimization (like SGD), Scientific Machine Learning (SciML) faces "stiff" loss landscapes dominated by differential equations. This paper explains why SciML requires a shift toward curvature-aware methods, adaptive sampling, and hybrid optimization strategies to achieve high-precision physical simulations.

What Problem Does This Paper Address?

In traditional ML (think image recognition), we have massive amounts of independent data points. We use Stochastic Gradient Descent (SGD) because it’s efficient to look at small "batches" of data at a time.

However, in SciML—specifically when training Physics-Informed Neural Networks (PINNs)—the "data" isn't just points; it’s a set of physical laws (Partial Differential Equations or PDEs). These laws couple every point in space and time together. This creates a loss landscape that is anisotropic and stiff: some directions are incredibly steep (high-frequency components), while others are nearly flat. Standard first-order optimizers like Adam often get "stuck" in these flat regions or oscillate wildly in the steep ones, failing to reach the high accuracy required for engineering and scientific applications.

Methods

The paper reviews a hierarchy of optimization techniques, categorizing them by how much "information" they use:

  • First-Order & Adaptive Methods: Standard tools like SGD, Momentum, and Adam. The authors explain Spectral Bias—the tendency of neural networks to learn low-frequency (smooth) parts of a solution long before they capture sharp gradients or high-frequency oscillations.
  • Second-Order & Quasi-Newton Methods: Techniques like L-BFGS and Hessian-free Newton-Krylov. These methods use "curvature" information (the Hessian) to navigate the stiff landscapes of SciML more effectively than Adam.
  • Adaptive Sampling: A key "meta-optimization" technique where the locations of training points (collocation points) are moved during training to areas where the PDE residual is highest.
  • Preconditioning: The authors discuss preconditioning in three spaces: Data-space (scaling inputs), Parameter-space (transforming gradients), and Function-space (modifying the loss geometry directly).

Datasets

As a tutorial and review paper, the authors utilize representative mathematical benchmarks rather than massive public datasets like ImageNet:

  • Binary Classification & Nonlinear Regression: Used to illustrate the basics of Empirical Risk Minimization (ERM) and spectral bias.
  • 1D Poisson Equation: A classic PDE used to demonstrate the "stiffness" of PINN training.
  • Synthetic Sinusoidal Functions: Used to visualize how networks struggle with multi-scale frequencies.
  • Tutorial Notebooks: The paper points to a GitHub repository containing practical implementations of these examples.

Key Insights and Ideas

  • The NTK Connection: The authors use the Neural Tangent Kernel (NTK) to prove that the difficulty of training a PINN is directly linked to the order of the differential operator ($p$). The condition number of the optimization problem scales with $\omega^{2p}$, meaning higher-order physics (like biharmonic equations) are exponentially harder to train.
  • Sampling is Optimization: In SciML, the "data" is chosen by the user. The paper argues that choosing where to sample the PDE is actually an optimization parameter. Moving points to "hard" regions of the domain can reduce the condition number of the NTK from ~200 down to ~13, making the problem much easier to solve.
  • The "Adam $\to$ L-BFGS" Handover: A major "aha moment" for practitioners is the hybrid strategy. Adam is excellent for the early, noisy phase of training to find the right "neighborhood," but L-BFGS is essential for the final "polishing" phase to satisfy physical constraints to high precision.

Key Results

  • Convergence Speed: In a 1D Poisson example, a hybrid Adam $\to$ L-BFGS strategy reached a loss of $10^{-4}$ in under 2,000 iterations, whereas standard Adam plateaued at a loss of $10^0$ even after 6,000 iterations.
  • Spectral Flattening: The paper demonstrates that second-order methods (like Gauss-Newton) effectively "flatten" the spectrum of the training kernel, narrowing the gap between the fastest- and slowest-learning components of the model.
  • Conditioning: Adaptive sampling was shown to make the level sets of the loss landscape more isotropic (circular rather than elongated ellipses), allowing for much larger, stable step sizes during training.

Limitations and Future Work

Despite the progress, the authors acknowledge several "open frontiers":

  • Stochastic Second-Order Methods: Most high-precision SciML currently relies on deterministic (full-batch) L-BFGS. Developing truly robust stochastic second-order methods that work with mini-batches remains a challenge.
  • Theoretical Gaps: While adaptive weighting and sampling work well in practice, we still lack a rigorous theoretical framework to choose the "optimal" weights for multi-objective physics losses.
  • Scalability: Many advanced preconditioning techniques (like multilevel or domain decomposition) are promising but difficult to implement efficiently for very large, deep neural networks.

Summary updated

Abstract

Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.
BibTeX
@misc{kopanicakova2026introduction,
  author = {Alena Kopaničáková and Elisa Riccietti},
  title = {{Introduction to optimization methods for training SciML models}},
  year = {2026},
  eprint = {2601.10222},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.10222},
}
Figure