On the approximation of rough functions with deep neural networks

Tim De Ryck, Siddhartha Mishra, Deep RaySeMA 2019 · DOI · OpenReview

Abstract

Deep neural networks and the ENO procedure are both efficient frameworks for approximating rough functions. We prove that at any order, the ENO interpolation procedure can be cast as a deep ReLU neural network. This surprising fact enables the transfer of several desirable properties of the ENO procedure to deep neural networks, including its high-order accuracy at approximating Lipschitz functions. Numerical tests for the resulting neural networks show excellent performance for approximating solutions of nonlinear conservation laws and at data compression.

Introduction

In physics and engineering, "rough functions"—those containing sharp edges, jumps, or discontinuities—are common, such as shock waves in gas dynamics. Standard approximation tools like linear interpolation or high-order polynomials often fail here. Linear interpolation loses accuracy near singularities, while polynomials suffer from "Gibbs oscillations," creating wild overshoots near discontinuities.

The Essentially Non-Oscillatory (ENO) procedure, developed in the 1980s, addresses this by adaptively choosing the "smoothest" stencil of points for interpolation. This paper establishes a rigorous mathematical bridge between this classical numerical algorithm and modern deep learning, proving that the ENO procedure is mathematically equivalent to a specific type of deep ReLU neural network.

Methods: The DeLENO Architecture

The authors demonstrate that the discrete logic of ENO stencil selection can be exactly represented by a deep neural network using Rectified Linear Unit (ReLU) activation functions. Because ReLUs are piecewise linear, they can mimic the "if-then" logic and undivided differences central to the ENO algorithm.

Architecture of the deep neural network (DeLENO) representing the ENO procedure.

The resulting "DeLENO" (Deep Learning ENO) networks frame stencil selection as a classification problem. ReLU networks partition the input space into polytopes, each corresponding to a specific stencil choice. This architecture allows the network to "learn" the ENO logic through standard optimization techniques like ADAM.

Illustration of stencil selection and sub-cell resolution (SR) for handling discontinuities.

The study also extends this to the ENO-SR (Sub-Cell Resolution) variant, which maintains second-order accuracy even for piecewise smooth functions by locating discontinuities within cells.

Results and Discussion

The DeLENO networks were evaluated across several benchmarks:

  • Classification Accuracy: Trained networks achieved 99.2% to 99.9% accuracy in matching classical ENO stencil selection.
  • High-Order Accuracy: DeLENO-4 matched the performance of 4th-order ENO, successfully suppressing oscillations near jumps in 1D benchmark functions.
  • Image Compression: In 2D tests on grayscale images, DeLENO achieved a compression rate of 0.996 while maintaining high visual fidelity, rivaling classical methods.
  • Physics Simulations: When applied to the Euler equations (e.g., Sod shock tube), the DeLENO-based solver captured shock waves with the precision of state-of-the-art numerical schemes.

Interestingly, the weights discovered through training differed from those in the theoretical proof, suggesting multiple ways for a network to implement the underlying ENO logic.

Conclusion

This work proves that classical numerical algorithms for rough functions can be "baked" into neural network architectures. While currently focused on 1D and tensor-product 2D grids, future work aims to extend these guarantees to high-dimensional, unstructured grids. Additionally, addressing the accuracy degradation observed on extremely fine grids in ENO-SR tests remains an open challenge for creating perfectly robust neural interpolators.

Summary updated

Abstract

Deep neural networks and the ENO procedure are both efficient frameworks for approximating rough functions. We prove that at any order, the ENO interpolation procedure can be cast as a deep ReLU neural network. This surprising fact enables the transfer of several desirable properties of the ENO procedure to deep neural networks, including its high-order accuracy at approximating Lipschitz functions. Numerical tests for the resulting neural networks show excellent performance for approximating solutions of nonlinear conservation laws and at data compression.
BibTeX
@inproceedings{ryck2019approximation,
  author = {Tim De Ryck and Siddhartha Mishra and Deep Ray},
  title = {{On the approximation of rough functions with deep neural networks}},
  booktitle = {SeMA},
  year = {2019},
  doi = {10.1007/s40324-022-00299-w},
}
Figure