Learning High-Dimensional Hilbert-Valued Functions With Deep Neural Networks From Limited Data

Ben Adcock, Simone Brugiapaglia, Nick C. Dexter, Sebastian MoragaAAAI 2021 · OpenReview

Abstract

This work establishes a theoretical and empirical bridge between deep learning and scientific computing. We demonstrate that Deep Neural Networks (DNNs) can be trained to approximate high-dimensional, Hilbert-valued functions—such as solutions to parametric partial differential equations (PDEs)—with efficiency comparable to state-of-the-art polynomial methods. While traditional methods excel on smooth functions, DNNs offer superior flexibility for complex, non-smooth problems and maintain numerical stability under noise.

Introduction

In scientific domains like engineering and physics, solving parametric PDEs is a fundamental challenge. These equations model system responses (e.g., heat distribution) based on high-dimensional input parameters (e.g., material properties). Two primary obstacles hinder progress: the curse of dimensionality, where data requirements grow exponentially with parameter count, and data scarcity, as each data point requires a costly numerical simulation.

While polynomial-based methods are the current standard, they often struggle with non-smooth functions. Deep learning presents a compelling alternative, yet a rigorous understanding of whether DNNs can be trained to match the accuracy of classical methods from limited data has been lacking. This paper addresses this gap by providing a "practical existence theorem" and extensive numerical validation.

Methodology

The authors employ a dual approach combining theoretical analysis with rigorous benchmarking:

  • Practical Existence Theorem: Moving beyond simple expressivity, Theorem 3.1 proves that a DNN can be learned via standard regularized training to achieve exponential convergence rates for holomorphic functions.
  • Hilbert-Valued Approximation: The framework maps high-dimensional parameters to elements of a Hilbert space, allowing the network to output entire fields or functions rather than scalar values.
  • Benchmarks: The study utilizes two high-dimensional scalar test functions ($d=8$) and a 2D diffusion equation with mixed boundary conditions. Ground truth for the PDE is generated via high-fidelity Finite Element Method (FEM) discretization (>22,000 degrees of freedom).
  • Training and Comparison: DNNs (ReLU, Leaky-ReLU, Tanh) are trained using the Adam optimizer with $l^2$-loss and compared against "best-in-class" polynomial-based compressed sensing methods (weighted and unweighted $\ell_1$-minimization).

Results and Discussion

Performance and the "Smoothness Gap"

Numerical experiments reveal that DNNs can achieve sample complexity similar to compressed sensing. However, for "perfectly" smooth functions, traditional polynomial methods currently outperform standard DNN training. This "smoothness gap" suggests that while the theoretical potential exists, standard optimizers like Adam may not yet fully exploit the network's capacity for these specific cases.

Comparison of DNN architectures against polynomial-based $\ell_1$-minimization for a smooth function.

Stability and Noise Robustness

A critical finding is the numerical stability of the trained models. DNNs handle measurement noise linearly; as Gaussian noise increases, the approximation error grows predictably rather than causing a model breakdown. This robustness is essential for real-world scientific applications where data is often imperfect.

Average relative $L^2$ error under varying levels of Gaussian noise for a ReLU 5x25 DNN.

Architecture and Efficiency

In parametric PDE tests, deeper and wider networks generally achieved lower training losses and faster convergence. Leaky-ReLU activations typically outperformed Tanh. Furthermore, the error bounds for the trained DNNs were found to be proportional to the discretization error of the underlying physical space, ensuring that the model's accuracy is well-aligned with the fidelity of the training data.

Conclusion

This research confirms that DNNs are viable, stable, and efficient tools for approximating high-dimensional Hilbert-valued functions. While an "optimization gap" remains for extremely smooth functions, the inherent flexibility of DNNs to handle discontinuities and their robustness to noise make them a powerful generalist tool for scientific computing. Future work will focus on closing the optimization gap through tailored loss functions and improving the approximation of gradients, which are vital for physical stress and flux analysis.

Summary updated

BibTeX
@inproceedings{adcock2021learning,
  author = {Ben Adcock and Simone Brugiapaglia and Nick C. Dexter and Sebastian Moraga},
  title = {{Learning High-Dimensional Hilbert-Valued Functions With Deep Neural Networks From Limited Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=pMkXkfG4aE},
}
Figure