Training Neural Networks with Derivative Supervision

54 concepts41 papers1 proposal

Standard supervised learning treats neural networks as point-wise curve fitters. Given an input, the network predicts an output, and the loss function penalizes the distance to the true label. While highly effective for standard regression, this approach treats each data point as an isolated island, ignoring how the target function behaves between those points.

If we instead supervise the network on both the function values and its derivatives with respect to the inputs, we force the model to match the local geometry—the slope, orientation, and curvature—of the target function. This paradigm, often mathematically framed around Sobolev spaces and referred to as Sobolev training, acts as a powerful inductive bias. As shown by Wojciech Czarnecki 0001 et al. in Sobolev Training for Neural Networks, incorporating target derivatives forces the model to capture local geometry, which dramatically improves sample efficiency and generalization. Instead of needing a dense grid of points to map a curve, the network learns the trajectory from a few points because it knows exactly which direction the curve is heading at each step. Qingtian Zhu et al. in Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives and Alessandro Trenta et al. in Learning and Transferring Physical Models through Derivatives demonstrate that this derivative-level supervision provides a richer informational signal, making models significantly more robust to data scarcity and allowing them to generalize better to unseen continuous coordinates.

The Geometry of Physical Landscapes

Why is this particularly transformative for scientific computing and physics-informed machine learning? In physics, the most critical observables are rarely the raw scalar potentials themselves; they are the derivatives of those potentials. For instance, in hyperelastic material modeling, we might want to learn an energy potential function, but the physical quantities we actually measure and care about are stresses (the first derivative of energy) and material stiffness (the second derivative).

When training neural network surrogates for these systems, optimizing solely on the scalar energy can lead to non-physical artifacts, such as unphysical oscillations or non-convex energy surfaces. Nikolaos N. Vlassis et al. in Geometric deep learning for computational mechanics Part I: anisotropic hyperelasticity, Dominik K. Klein et al. in Neural networks meet hyperelasticity: A guide to enforcing physics, and Karl A. Kalina et al. in Neural networks meet anisotropic hyperelasticity: A framework based on generalized structure tensors and isotropic tensor functions address this by training networks directly on the derivatives of the energy potential. By minimizing errors on the resulting stress fields, the learned energy landscape remains smooth, thermodynamically valid, and physically consistent.

Furthermore, Alex Beatson et al. in Learning Composable Energy Surrogates for PDE Order Reduction demonstrate that capturing the exact shape of this energy landscape through forces and stiffness is not just an optional enhancement—it is a strict requirement for ensuring that downstream gradient-based solvers remain stable during deployment. This principle extends across diverse physical domains, from modeling finite strain magneto-elasticity (Karl A. Kalina et al. in Neural network-based multiscale modeling of finite strain magneto-elasticity with relaxed convexity criteria) to capturing the intricate warping of 3D-printed geometrically exact beams (Dominik K. Klein et al. in Physics-augmented neural networks for constitutive modeling of hyperelastic geometrically exact beams).

Preventing Spurious Minima in Downstream Optimization

Beyond physical realism, derivative supervision solves a fundamental engineering headache: the fragility of optimization loops that rely on neural network surrogates. If a neural network is trained only on value outputs, its learned function may contain tiny, high-frequency wiggles. While these wiggles might not significantly affect the value accuracy, their derivatives can be wildly incorrect. If you pass this surrogate to a gradient-based optimizer or a controller, the optimizer will get trapped in these artificial local minima.

By explicitly penalizing discrepancies in the predicted derivatives or Jacobians, we smooth out these spurious artifacts. Dingcheng Luo et al. in SOUPy: Stochastic PDE-constrained optimization under high-dimensional uncertainty in Python and Andrew W. Rosemberg et al. in Sobolev Training of End-to-End Optimization Proxies demonstrate this mechanism in action for PDE-constrained optimization and end-to-end optimization proxies. By forcing the network to learn the true sensitivities of the system, the surrogate accurately maps the local geometry of the solution space, drastically reducing constraint violations and preventing optimization loops from getting trapped.

As Yuan Qiu et al. in Derivative-enhanced Deep Operator Network point out, this derivative-augmented loss ensures that the model correctly predicts how outputs change in response to input perturbations, making it highly robust for downstream tasks like Bayesian inference. It also allows surrogates to accurately capture complex dual relationships, such as predicting Lagrange multipliers in entropy-based closure problems (Steffen Schotthöfer et al. in Structure Preserving Neural Networks: A Case Study in the Entropy Closure of the Boltzmann Equation).

Mathematical Foundations and Optimization Benefits

Intuitively, adding derivative constraints sounds like it might make the learning problem harder or exponentially increase the complexity of the model's hypothesis class. However, theoretical insights prove the opposite. Yahong Yang et al. in Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives established nearly optimal VC-dimension and pseudo-dimension bounds for deep neural network derivatives, proving that derivative constraints do not exponentially increase hypothesis complexity. Consequently, practitioners can incorporate physical laws or gradient information without requiring a disproportionately larger volume of training data to achieve generalization.

In fact, derivative supervision can actually accelerate training. Jose Blanchet et al. in Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent identify a phenomenon known as "Sobolev Implicit Acceleration," demonstrating that incorporating higher-order derivative penalties into the loss function improves the condition number of the optimization problem, leading to faster gradient descent convergence. They also show that this can be achieved even when ground-truth derivative data is unavailable by applying integration by parts to noisy function observations.

When designing architectures for Sobolev training, the choice between depth and width matters. Yahong Yang et al. in Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss reveal that while deeper networks offer superior approximation power ("super-convergence"), they require significantly more data to generalize under a Sobolev loss. Conversely, wider networks provide a safer, more stable alternative when data is scarce. For complex operator learning tasks, the author in DeepONet for Solving Nonlinear Partial Differential Equations with Physics-Informed Training derives generalization error bounds in Sobolev norms, confirming that evaluating losses on both solution values and derivatives provides a rigorous mathematical foundation for solving non-linear PDEs.

High-Fidelity Continuous Representations

The benefits of Sobolev training are especially pronounced in continuous neural representations—such as Implicit Neural Representations (INRs) and neural operators—where the network acts as a continuous field.

In chaotic or turbulent physical systems, energy transfers across multiple spatial and temporal scales. Zongyi Li et al. in Learning Chaotic Dynamics in Dissipative Systems show that incorporating Sobolev norms into the loss function of neural operators is essential for capturing these high-frequency components and multiscale dynamics. Similarly, in computational general relativity, Sandeep Suresh Cranganore et al. in Einstein Fields: A Neural Perspective To Computational General Relativity use Sobolev supervision (including Jacobians and Hessians) to train INRs that compress massive spacetime simulations, achieving orders-of-magnitude higher accuracy in derived physical quantities like curvature and wave strains compared to traditional grid-based methods.

This capacity to capture exact gradients is vital for manufacturing and control. For instance, M. Manav et al. in MeltpoolINR: predicting temperature field, melt pool geometry and their rate of change in laser powder bed fusion utilize Sobolev training in laser powder bed fusion modeling to accurately capture spatial temperature gradients and cooling rates, which are critical for predicting material defects. To guarantee mathematical rigor across the entire domain, Johannes Müller et al. in Notes on Exact Boundary Values in Residual Minimisation show that combining Sobolev-norm residual minimization with exact boundary conditions turns the training loss into an honest indicator of the true solution error.

Finally, to maximize the precision of these gradients in high-dimensional non-linear PDEs, Chuwei Wang et al. in Is $L^2$ Physics Informed Loss Always Suitable for Training Physics Informed Neural Network? demonstrate that optimizing for maximum residuals ($L^\infty$ loss) rather than average residuals ($L^2$ loss) significantly enhances gradient accuracy. This structural consistency even benefits reinforcement learning; Baptiste Debes et al. in Distributional value gradients for stochastic environments show that a Sobolev inductive bias—where an agent's value function gradients are aligned with target gradients—significantly enhances sample efficiency in noisy, stochastic environments.

Key Foundational Work

Learning Composable Energy Surrogates for PDE Order Reduction (2020) — contributed 2 concept elements to this direction.
Geometric deep learning for computational mechanics Part I: anisotropic hyperelasticity (2020) — contributed 2 concept elements to this direction.
Global Convergence of Sobolev Training for Overparameterized Neural Networks (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sobolev_training_e27aa55a,
  title = {Sobolev training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sobolev-training/rdr_e27aa55aa7ff458b955fae6d1daf8111},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sobolev_training_e27aa55a_v1,
  title = {Sobolev training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sobolev-training/rdr_e27aa55aa7ff458b955fae6d1daf8111/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake