Sobolev acceleration for neural networks

Jong Kwon Oh, Hanbaek Lyu, Hwijae SonSubmission to ICLR 2026 · Rejected

TL;DR

While empirical studies have long shown that penalizing derivative errors alongside standard value errors (Sobolev training) speeds up neural network optimization, the mathematical reason why has remained unexplained. This paper presents the first rigorous theoretical framework proving that Sobolev training fundamentally improves the loss landscape's Hessian conditioning and accelerates gradient flow in shallow ReLU networks. Furthermore, the authors demonstrate that this acceleration robustly carries over to modern deep learning workloads, including denoising autoencoders and diffusion models.

What Problem Does This Paper Address?

In standard neural network regression, models minimize a conventional $L^2$ loss (mean squared error of predicted values). In many scientific and physical problems, target derivatives are known or can be approximated. Minimizing the Sobolev norm (matching both function values and derivatives, e.g., $H^1$ or $H^2$ norms) often accelerates empirical convergence and boosts generalization—a phenomenon called Sobolev acceleration.

However, existing theoretical optimization tools (such as standard Neural Tangent Kernel analyses) treated derivatives as separate output targets rather than inherently tied differential operators, failing to explain why or by how much training accelerates.

Methods

The authors analyze the population dynamics under a standard student–teacher framework with spherical Gaussian inputs $\mathcal{N}(0, I_{d \times d})$:

  • Landscape Conditioning Analysis ($K=1$ ReLU Node): They derive exact, closed-form formulas for the population Hessian of both the standard $L^2$ loss ($\nabla_w^2 \mathcal{L}$) and the $H^1$ Sobolev loss ($\nabla_w^2 \mathcal{H}$). By evaluating extreme eigenvalues, they directly compute condition numbers $\kappa(\nabla^2 \mathcal{H})$ vs. $\kappa(\nabla^2 \mathcal{L})$.
  • Gradient Flow Dynamics ($K \ge 1$ Hidden Units):
  • For $K=1$, they track the parameter error $V(w) = \|w - w^*\|_2^2$ under continuous gradient flow.
  • For higher-order derivatives ($H^2$), they analyze networks with squared ReLU activations $\sigma(w^\top x)^2$.
  • For general two-layer networks ($K > 1$) with orthogonal teacher weights, they linearize the dynamical system around the ground truth to prove convergence rates.
  • Spectral Implementation for Deep Architectures: To apply $H^1$ Sobolev training efficiently to image tasks without explicit analytical labels, they employ Fast Fourier Transforms (FFT) to penalize high-frequency derivative errors in the spectral domain.

Datasets

The authors validate their framework across synthetic benchmarks and large-scale vision datasets:

  • Synthetic / Benchmark Functions: Multidimensional standard Gaussian inputs; Ackley benchmark function; 1D multiscale oscillating target $f(x) = x + \sin(2\pi x^4)$.
  • CelebA-HQ: Downsampled to $128 \times 128$ resolution for training a Denoising Diffusion Probabilistic Model (DDPM / DDIM).
  • ImageNet: Resized to $3 \times 64 \times 64$ for training a ResNet-18 denoising autoencoder.

No new benchmarks were introduced.

Key Insights and Ideas

  • Condition Number Compression: Doubling an $L^2$ loss simply scales all Hessian eigenvalues uniformly, leaving the condition number $\kappa = \lambda_{\max}/\lambda_{\min}$ unchanged. In contrast, $H^1$ Sobolev training drastically inflates $\lambda_{\min}$ while barely altering $\lambda_{\max}$, strictly decreasing the condition number: $$\kappa(\nabla_w^2 \mathcal{H}) = \frac{1}{1 - 4\alpha\sin^2\theta} < \frac{1}{1 - 3\alpha\sin^2\theta} = \kappa(\nabla_w^2 \mathcal{L})$$ Matching gradients introduces parameter update directions that are "nearly uncorrelated" with zeroth-order mismatches.

  • Larger Convexity Basin: Sobolev training expands the region of strict convexity around the global optimum (the basin of attraction expands from $\sin\theta < \frac{\pi\|w\|}{2\|w^*\|}$ under $L^2$ to $\sin\theta < \frac{2\pi\|w\|}{3\|w^*\|}$ under $H^1$), eliminating spurious local minima and saddle points present in $L^2$ training.

  • Doubled Local Convergence Speed: For general two-layer networks ($K > 1$), local linearization around the ground truth reveals that the linearized system matrix under $H^1$ gradient flow is exactly twice that of $L^2$ flow ($\dot{T}_{H^1} = -2M T_{H^1}$ vs. $\dot{T}_{L^2} = -M T_{L^2}$), provably doubling convergence rates.

Key Results

  • Empirical Risk Minimization: SGD experiments with varying batch sizes and learning rates confirm lower condition numbers and accelerated convergence of parameter errors for single ReLU models.
  • Architectural Robustness: Sobolev acceleration consistently reduces epochs-to-convergence across multiple activation functions (ReLU, GeLU, Tanh, Sine) and representations (Fourier Feature Networks, SIREN).
  • Diffusion Models (CelebA-HQ): Training a U-Net DDPM with an $H^1$ loss accelerates the reduction of Fréchet Inception Distance (FID) scores and produces sharper image samples. While the isolated $H^1$ loss evaluation is $\approx 2.3\times$ slower than $L^2$ (0.028s vs. 0.012s), the overall per-step time increases by only $\sim 1.7\%$ (1.003s vs. 0.986s), yielding a net wall-clock speedup to target FID.

Limitations and Future Work

  • Stylized Theoretical Setting: The mathematical proofs rely heavily on standard Gaussian input distributions, two-layer shallow networks, and exact student–teacher parameter matching.
  • High-Order Derivatives: Calculating exact higher-order derivatives for deep networks via automatic differentiation can become computationally intractable; practical scalability currently hinges on numerical or spectral approximations.
  • Future Work: Extending the gradient flow dynamical analysis to deep, multi-layer architectures and arbitrary non-Gaussian input distributions remains an open theoretical challenge.

Summary updated

Abstract

$\textit{Sobolev training}$, which integrates target derivatives into the loss functions, has been shown to accelerate convergence and improve generalization compared to conventional $L^2$ training. However, the underlying mechanisms of this training method remain incompletely understood. In this work, we show that Sobolev training provably accelerates the convergence of Rectified Linear Unit (ReLU) networks and quantify such `Sobolev acceleration' within the student--teacher framework. Our analysis builds on an analytical formula for the population gradients and Hessians of ReLU networks under centered spherical Gaussian input. Extensive numerical experiments validate our theoretical findings and show that the benefits of Sobolev training extend to modern deep learning tasks, including diffusion models.
BibTeX
@misc{oh2025sobolev,
  author = {Jong Kwon Oh and Hanbaek Lyu and Hwijae Son},
  title = {{Sobolev acceleration for neural networks}},
  year = {2025},
  eprint = {2509.19773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.19773},
}

Loading related content...

Figure