Physics-Informed Neural Networks (PINNs) have emerged as a powerful alternative to traditional numerical solvers, but they suffer from a fundamental "Achilles' heel": the distribution of training points. In standard PINNs, collocation points—the coordinates where the model evaluates the physical residual—are typically sampled uniformly or on a fixed grid.
However, physical phenomena are rarely uniform. Features like shock waves, sharp gradients, and singularities occupy only a tiny fraction of the spatial domain but dictate the entire system's behavior. When a model spends 99% of its computational budget on "empty" regions where the solution is smooth, it lacks the signal necessary to resolve these critical localized features. Recent research has shifted toward Adaptive Sampling, a family of techniques that dynamically relocate training points to where the physics is hardest to learn.
Residuals as Navigational Beacons
The most intuitive way to adapt sampling is to treat the model’s own failure as a guide. If the PDE residual is high in a specific region, it indicates that the current network parameters do not yet satisfy the physical constraints there.
Researchers like Yanjie Wang have formalized this through Maximum Residual-based continuous Distribution (MRD), which uses a secondary "navigator" network to map the residual landscape and push points toward high-error zones Navigating PINNs via maximum residual-based continuous distribution. Similarly, Arka Daw and Paris Perdikaris introduced the Retain-Resample-Release (R3) algorithm, which treats training points as a dynamic population that accumulates in high-residual regions while being "released" from areas where the physics is already well-resolved Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling.
Generative Error Hunting
While simple residual-based methods work well in low dimensions, they struggle as the complexity of the domain increases. A more robust approach involves training a generative model to "hunt" for errors.
In the Adversarial Adaptive Sampling (AAS) framework developed by Kejun Tang and Xiaoliang Wan, the training process becomes a min-max game. A generative network attempts to find points that maximize the PDE residual, while the PINN attempts to minimize it Adversarial Adaptive Sampling: Unify PINN and Optimal Transport for the Approximation of PDEs. This effectively smooths the residual profile across the domain. Xiaoliang Wan further extended this using bounded KRnets (a type of normalizing flow) to handle high-dimensional problems where uniform sampling becomes exponentially inefficient Adaptive importance sampling for Deep Ritz.
Beyond Residuals: Geometric and Physical Indicators
Relying solely on the raw residual can be misleading; a high residual doesn't always mean a high approximation error. To solve this, researchers are looking at the "stiffness" or geometry of the solution space.
- Hessian-Based Sampling: Antoine Caradot and colleagues proposed hessian-RAD, which uses the second-order derivative (Hessian) of the residuals to place points. By focusing on regions of high curvature, they provide a theoretically grounded way to capture sharp transitions that first-order methods might miss Provably Accurate Adaptive Sampling for Collocation Points in Physics-Informed Neural Networks.
- Energy Dissipation: For thermodynamic systems, Chunyan Li introduced EDRAS, which uses the physical energy dissipation rate as a sampling guide. This ensures the model resolves the interfaces where the most significant physical transformations occur, rather than just mathematical artifacts Energy Dissipation Rate Guided Adaptive Sampling for Physics-Informed Neural Networks.
Borrowing from Classical Numerical Analysis
The idea of moving points to follow "interesting" features is not new; it is the core of "moving mesh" methods in traditional Computational Fluid Dynamics (CFD). Modern research is now "neuralizing" these concepts.
Yangtao Deng developed MS-PINN, which uses a dedicated MMPDE-Net to predict how collocation points should migrate over time, mimicking a mesh that deforms to track a shock wave MMPDE-Net and Moving Sampling Physics-informed Neural Networks Based On Moving Mesh Method. Similarly, the Neural Galerkin schemes proposed by Yuxiao Wen and Benjamin Peherstorfer evolve an ensemble of particles that naturally cluster around traveling waves and fronts, providing orders-of-magnitude higher accuracy than static sampling Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes.
By treating the placement of training data as a learnable, dynamic process rather than a static preprocessing step, these adaptive strategies allow neural solvers to tackle "stiff" and high-dimensional physics problems that were previously considered the exclusive domain of supercomputer-scale classical simulations.