Sobolev acceleration for neural networks
Lists
Only you see this
Abstract
Sobolev training for neural networks, a technique that integrates target derivatives into the training process, has demonstrated significantly faster convergence towards lower test errors when compared to conventional loss functions. However, to date, the effect of this training has not been understood comprehensively. This paper presents analytical evidence that Sobolev training accelerates the convergence of rectified linear unit (ReLU)-networks in the student-teacher framework. The analysis builds upon the analytical formula for the population gradients of ReLU networks with centered spherical Gaussian input. Further, numerical examples were considered to show that the results may be extended to multi-layered neural networks with various activation functions and architectures. Finally, we propose the use of Chebyshev spectral differentiation as a solution to approximate target derivatives and address prior limitations on using approximated derivatives. Overall, this study contributes to a deeper understanding of the dynamics of ReLU networks in the student-teacher setting and highlights the convergence acceleration achieved through Sobolev training, known as Sobolev acceleration.
BibTeX
@misc{son2025sobolev,
author = {Hwijae Son},
title = {{Sobolev acceleration for neural networks}},
year = {2025},
eprint = {2509.19773},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2509.19773},
}