Sobolev acceleration for neural networks

Jong Kwon Oh, Hanbaek Lyu, Hwijae Son

Abstract

Sobolev training, which integrates target derivatives into the loss functions, has been shown to accelerate convergence and improve generalization compared to conventional $L^2$ training. However, the underlying mechanisms of this training method remain only partially understood. In this work, we present the first rigorous theoretical framework proving that Sobolev training accelerates the convergence of Rectified Linear Unit (ReLU) networks. Under a student-teacher framework with Gaussian inputs and shallow architectures, we derive exact formulas for population gradients and Hessians, and quantify the improvements in conditioning of the loss landscape and gradient-flow convergence rates. Extensive numerical experiments validate our theoretical findings and show that the benefits of Sobolev training extend to modern deep learning tasks.

Introduction

Standard neural network training typically relies on the $L^2$ loss, which measures the point-wise difference between the network's output and the target value. However, in many scientific and engineering fields—such as fluid dynamics or physics-informed machine learning—we often have access to the target's derivatives (gradients) as well. "Sobolev training" incorporates these derivatives into the loss function, effectively matching the "shape" of the target function rather than just its values.

While empirical evidence has long suggested that this approach improves accuracy and speeds up training—a phenomenon termed Sobolev acceleration—the theoretical justification remained elusive. This paper bridges that gap by providing a rigorous mathematical foundation for why derivative information makes the optimization problem fundamentally easier to solve.

Theoretical Framework: The Student-Teacher Model

To analyze the dynamics of Sobolev training, the authors employ a student-teacher framework. In this setup, a "teacher" network represents the ground truth, and a "student" network attempts to learn its parameters. By focusing on shallow ReLU networks and assuming Gaussian input distributions, the researchers derived exact analytical formulas for the population Hessian—the matrix of second-order derivatives of the loss function.

The "Conditioning" Secret

The speed of gradient-based optimization is largely governed by the "condition number" of the Hessian, which is the ratio of its largest eigenvalue to its smallest. A high condition number indicates a "narrow valley" in the loss landscape, making it difficult for the optimizer to converge. The authors prove that Sobolev ($H^1$) training significantly increases the minimum eigenvalue of the Hessian while leaving the maximum eigenvalue relatively unchanged. This "squeezes" the condition number, creating a more spherical loss landscape that is significantly easier for gradient descent to navigate.

Furthermore, Sobolev training creates larger "basins of attraction"—regions where the loss function is strictly convex. This prevents the optimizer from getting stuck in the flat regions or saddle points that typically slow down standard $L^2$ training.

Empirical Results and Acceleration

The theoretical predictions were validated across a wide range of architectures and activation functions. Whether using ReLU, Leaky-ReLU, GeLU, or periodic activations like Sine (SIREN), Sobolev training consistently reached lower error thresholds in fewer iterations.

Convergence of training loss across different activation functions. Sobolev training ($H^1$ and $H^2$) demonstrates significant acceleration compared to standard $L^2$ training.

Beyond simple MLPs, the acceleration effect was observed in Fourier Feature Networks and SIRENs, suggesting that the benefits of Sobolev training are architecture-agnostic.

Qualitative Performance and Generalization

The benefits of Sobolev training extend beyond speed. In denoising and reconstruction tasks, models trained with derivative information exhibit superior generalization. For instance, in autoencoder experiments on ImageNet and MNIST, Sobolev-trained models produced reconstructions that were noticeably clearer and free of the blurry artifacts or color shifts often seen in $L^2$-trained models.

Reconstruction of noisy digits. Sobolev-trained models ($H^1$) produce clearer results with fewer artifacts than those trained with standard $L^2$ loss.

In modern generative tasks, such as Diffusion Models (DDPM) trained on CelebA-HQ, Sobolev training led to faster convergence of the Fréchet Inception Distance (FID) score. Remarkably, this performance boost comes with negligible computational overhead. Experiments showed that memory usage and per-epoch training time for $H^1$ training were within 1% of standard $L^2$ training.

Conclusion and Future Work

This work provides the first rigorous proof of Sobolev acceleration in neural networks, identifying improved Hessian conditioning as the primary driver of faster convergence. While the current mathematical proofs are focused on shallow networks and Gaussian inputs, the empirical results strongly suggest these principles hold for deep architectures and complex, real-world datasets. Future research will look to extend this theoretical framework to multi-layer networks and explore the interaction between Sobolev training and various forms of data noise.

Summary updated

Abstract

Sobolev training, which integrates target derivatives into the loss functions, has been shown to accelerate convergence and improve generalization compared to conventional $L^2$ training. However, the underlying mechanisms of this training method remain only partially understood. In this work, we present the first rigorous theoretical framework proving that Sobolev training accelerates the convergence of Rectified Linear Unit (ReLU) networks. Under a student-teacher framework with Gaussian inputs and shallow architectures, we derive exact formulas for population gradients and Hessians, and quantify the improvements in conditioning of the loss landscape and gradient-flow convergence rates. Extensive numerical experiments validate our theoretical findings and show that the benefits of Sobolev training extend to modern deep learning tasks.
BibTeX
@misc{oh2025sobolev,
  author = {Jong Kwon Oh and Hanbaek Lyu and Hwijae Son},
  title = {{Sobolev acceleration for neural networks}},
  year = {2025},
  eprint = {2509.19773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.19773},
}
Figure