Physics-Informed Neural Networks (PINNs) have emerged as a powerful tool for solving partial differential equations (PDEs) by embedding physical laws directly into a neural network's loss function. To train a PINN, we evaluate how well the network satisfies these physical laws at a discrete set of coordinates known as collocation points.
In standard implementations, these collocation points are distributed uniformly or on a fixed grid across the domain. However, physical systems are rarely uniform. Complex phenomena—such as shock waves, turbulent eddies, phase transitions, or rare transitions between chemical states—are highly localized. They occupy tiny fractions of the spatial domain but dictate the entire system's dynamics. As noted by Marc Anton Finzi et al. in A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks, static sampling methods struggle to resolve these localized features where solutions change rapidly. Uniform sampling wastes the model's capacity on flat, uninformative regions while starving the complex regions of the resolution they need to converge.
To overcome this bottleneck, a major research direction focuses on adaptive collocation sampling. Instead of keeping points fixed, these strategies dynamically relocate, inject, or sample points where the model struggles most. This approach acts as the mesh-free equivalent of Adaptive Mesh Refinement (AMR) in classical computational fluid dynamics.
Residual-Driven Refinement: Hunting for Error
The most intuitive indicator of where a model needs more resolution is the PDE residual. The residual measures how severely the neural network's current predictions violate the governing physical equations. If the residual is zero, the physics are perfectly satisfied; if it is high, the model is failing to capture the local dynamics.
In frameworks like Residual-Based Adaptive Refinement (RAR), the training loop periodically evaluates the domain to identify high-residual zones and injects new collocation points into those areas. For example, Qianwei Dai et al. in A Data Assimilation Methodology to Analyze the Unsaturated Seepage of an Earth–Rockfill Dam Using Physics-Informed Neural Networks Based on Hybrid Constraints and Biao Yuan et al. in Physics-Informed Neural Networks for Fast 3D Consolidation Prediction: A Surrogate Modelling Application demonstrate that targeting high-residual regions allows networks to resolve sharp gradients in complex engineering simulations without requiring dense global grids.
Beyond simply improving local accuracy, this adaptive relocation fundamentally alters the optimization landscape. As Alena Kopaničáková explains in Introduction to optimization methods for training SciML models, dynamically shifting collocation points to high-residual regions functions as a form of meta-optimization. It reduces the condition number of the training kernel and makes the loss landscape more isotropic. This stabilization allows practitioners to use larger, more stable learning rates, accelerating overall convergence.
Higher-Order Guidance: Curvature and Hessians
While the magnitude of the residual tells us where the model is currently wrong, it does not reveal how volatile the error surface is. To track highly dynamic features like moving shocks, researchers look to higher-order derivatives of the residual to capture its curvature.
In Investigating Guiding Information for Adaptive Collocation Point Sampling in PINNs, He Wang et al. introduce a residual curvature-based sampling mechanism. By using the mixed second derivative of the PDE residual ($Base_{xt}$), the algorithm identifies regions where the error gradient changes rapidly across both spatial and temporal dimensions. This allows the model to anticipate and concentrate its limited point budget on evolving, high-complexity features.
Taking a more theoretical approach, Antoine Caradot et al. in Provably Accurate Adaptive Sampling for Collocation Points in Physics-Informed Neural Networks prove that placing collocation points according to the Hessian (second-order derivative) of the residuals minimizes the mathematical approximation error bound of the physics-informed loss. By targeting regions of high curvature where the physics constraints are most difficult to satisfy, their method achieves tighter error bounds and faster convergence than uniform or purely gradient-based sampling.
Probabilistic and Generative Sampling
Another powerful paradigm re-frames adaptive sampling as a probability estimation problem. Instead of adding points deterministically, these methods treat either the squared residual or the predicted physical quantity as an unnormalized probability density function (PDF) from which new points are drawn.
In Deep adaptive sampling for surrogate modeling without labeled data, Kejun Tang et al. train a deep generative model to treat the squared PDE residual as a PDF. The generative model adaptively "hunts" for high-error regions, drawing new batches of training points from the areas where the physics-based error is highest. Similarly, when dealing with unbounded domains or high-dimensional spaces, Li Zeng et al. in Adaptive deep density approximation for fractional Fokker-Planck equations utilize Normalizing Flows to sample training points directly from the model's current estimated density, ensuring computational resources are not wasted on regions with negligible probability mass.
This density-driven approach is equally effective when the physical signal itself is highly localized. In Truly Mesh-free Physics-Informed Neural Networks, Fabricio Arend Torres et al. use Markov Chain Monte Carlo (MCMC) methods to sample collocation points directly from the predicted physical density. By concentrating evaluation points where the physical signal is strongest, they achieved higher accuracy in 3D fluid simulations using just 512 points than standard uniform methods achieved with over 30,000 points. For quantum systems, Elena Orlova demonstrates in Deep Stochastic Mechanics that trajectory-based sampling naturally concentrates computational effort on high-density regions of the wave function, bypassing the curse of dimensionality inherent to uniform grids.
Continuous Relocation and Adversarial Tracking
Rather than discarding old points and resampling a new batch from scratch, some frameworks treat collocation points as dynamic particles that move continuously across the domain during training.
In Navigating PINNs via maximum residual-based continuous distribution, Yanjie Wang et al. propose a Maximum Residual-based continuous Distribution (MRD) framework. Here, a secondary neural network maps the residual landscape, and collocation points perform gradient ascent directly on this learned error surface. This continuous movement allows the points to precisely track and follow sharp features, like phase transitions, as they evolve over training iterations.
Alternatively, this can be framed as an adversarial game. Chuwei Wang et al. in Is $L^2$ Physics Informed Loss Always Suitable for Training Physics Informed Neural Network? demonstrate that standard $L^2$ loss lacks stability for high-dimensional non-linear PDEs. To fix this, the authors propose a gradient-based adversarial sampling method. By performing iterative updates on the input coordinates to maximize the physics-informed residual, they force the network to minimize an $L^\infty$ loss, adaptively focusing on the most challenging regions of the input space.
Managing the Point Budget Efficiently
While adaptive sampling drastically improves sample efficiency, it introduces a new challenge: computational overhead. Evaluating residuals over massive auxiliary grids to find out where to add points can become a bottleneck that consumes more time than the actual network training.
To address this, Arka Daw et al. developed the Retain-Resample-Release ($R^3$) sampling strategy in Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling. Instead of performing dense grid searches, the algorithm dynamically manages a fixed point budget through a three-part lifecycle:
- Retain: Points with high residuals are kept in the training set to force the optimizer to confront them.
- Resample: Regions where the error has been successfully minimized are replaced with fresh uniform samples to maintain a global representation of the domain.
- Release: Points are let go once their local error drops below a specified threshold.
By avoiding dense grid evaluations and recycling points dynamically, such memory-efficient management strategies ensure that adaptive collocation sampling remains computationally practical while solving highly non-linear, complex physical systems.
Key Foundational Work
VarNet: Variational Neural Networks for the Solution of Partial Differential Equations (2019) — contributed 2 concept elements to this direction.