Depth separation for reduced deep networks in nonlinear model reduction: Distilling shock waves in nonlinear hyperbolic problems

Donsub Rim, Luca Venturi, Joan Bruna, Benjamin PeherstorferarXiv 2020 · arXiv

Abstract

Classical model order reduction (MOR) techniques often struggle with nonlinear hyperbolic partial differential equations (PDEs), particularly those involving moving discontinuities or shock waves. This paper introduces Reduced Deep Networks (RDNs), a novel architecture designed to overcome the limitations of linear subspace approximations. By leveraging the power of depth, RDNs can represent solution manifolds of hyperbolic problems with exponentially fewer parameters than traditional linear models. We demonstrate that while linear methods suffer from slow convergence due to the Kolmogorov N-width, RDNs achieve high precision with only $O(|\log(\epsilon)|)$ degrees of freedom.

Introduction

In complex engineering domains such as aerodynamics, climate modeling, and fluid dynamics, simulating large-scale systems of PDEs is computationally prohibitive. Model Order Reduction (MOR) aims to alleviate this by creating simplified, low-dimensional representations of these systems. Traditionally, MOR relies on low-rank approximations, where the solution is projected onto a fixed linear basis.

While highly effective for parabolic problems (like heat diffusion), these linear methods fail for hyperbolic problems characterized by transport and shock waves. Because shocks are sharp features that shift in space and time, a fixed linear basis requires an impractical number of components to capture the feature at every possible location. This paper addresses this fundamental bottleneck by proposing a deep learning-based framework that utilizes coordinate transformations to "track" these moving features.

The Failure of Linear Subspaces

The efficiency of linear MOR is governed by the Kolmogorov N-width, which measures how well a set of functions can be approximated by a linear subspace of dimension $N$. For hyperbolic problems, the N-width decays very slowly. Mathematically, representing a moving step function (a simple shock) using a linear combination of fixed basis functions leads to the Gibbs phenomenon or requires a basis size that grows linearly with the desired resolution. This inefficiency makes real-time simulation of nonlinear transport problems nearly impossible with classical tools.

Proposed Method: Reduced Deep Networks (RDNs)

The authors propose Reduced Deep Networks (RDNs) as a solution. RDNs are not standard black-box neural networks; they are specifically structured to generalize and improve upon classical reduced models.

Key Architectural Components

  1. Low-Rank Weight Matrices: To maintain the efficiency of MOR, RDNs utilize weight matrices that are themselves low-rank. This ensures the total number of degrees of freedom remains small.
  2. Compositional Structure: The network is built as a sequence of layers where each layer performs a specific task, such as a nonlinear coordinate transformation (transport) or an approximation of the initial state.
  3. Approximate Inverses: A standout feature of the RDN is the inclusion of layers designed to approximate the inverse of a function. Using a bisection-style algorithm implemented via ReLU activations, the network can "undo" complex transport effects, effectively simplifying the representation of the shock wave.
  4. Depth as a Coordinate Transform: The fundamental "aha moment" of the paper is that depth allows the network to learn a sequence of coordinate transformations. Instead of approximating a moving shock in a fixed coordinate system, the deep layers effectively "flatten" or "track" the shock, making the final approximation task trivial.

Theoretical Framework and Results

The paper provides rigorous proofs for depth separation, showing that deep networks are fundamentally more capable than shallow or linear models for this class of problems.

The "Sharply Convective" Class

The authors define a new class of functions that encapsulate the difficulty of shocks. They prove that for this class:

  • Linear Models: The approximation error decays algebraically (slowly), requiring a massive number of parameters for high accuracy.
  • RDNs: The approximation error $\epsilon$ decays geometrically (fast), requiring only $O(|\log(\epsilon)|)$ degrees of freedom.

Benchmark Evaluations

The approach was evaluated on two fundamental mathematical benchmarks:

  • The Color Equation: A variable-speed transport equation used to test the model's ability to handle features moving at heterogeneous rates.
  • Burgers’ Equation: A classic nonlinear PDE where smooth initial conditions eventually "break" into sharp shocks. The RDN successfully resolved the singularities where characteristic lines collide, a feat that typically breaks simpler transport-based models.

Conclusion and Future Work

This research establishes a theoretical foundation for why deep learning succeeds where classical linear MOR fails in the context of physics-informed simulations. By using depth to perform nonlinear coordinate transformations, RDNs bypass the limitations of the Kolmogorov N-width.

However, challenges remain. The current work is primarily a constructive existence proof; training these specific RDN architectures using standard backpropagation can be difficult due to the rigid constraints on weight matrices. Furthermore, while the proofs are robust for one-dimensional problems, extending these results to 2D and 3D fluid dynamics remains a critical frontier for future research. Despite these hurdles, RDNs represent a significant step toward efficient, high-fidelity reduced-order modeling of the world's most complex physical systems.

Summary updated

Abstract

Classical reduced models are low-rank approximations using a fixed basis designed to achieve dimensionality reduction of large-scale systems. In this work, we introduce reduced deep networks, a generalization of classical reduced models formulated as deep neural networks. We prove depth separation results showing that reduced deep networks approximate solutions of parametrized hyperbolic partial differential equations with approximation error $\epsilon$ with $\mathcal{O}(|\log(\epsilon)|)$ degrees of freedom, even in the nonlinear setting where solutions exhibit shock waves. We also show that classical reduced models achieve exponentially worse approximation rates by establishing lower bounds on the relevant Kolmogorov $N$-widths.
BibTeX
@misc{rim2020depth,
  author = {Donsub Rim and Luca Venturi and Joan Bruna and Benjamin Peherstorfer},
  title = {{Depth separation for reduced deep networks in nonlinear model reduction: Distilling shock waves in nonlinear hyperbolic problems}},
  year = {2020},
  eprint = {2007.13977},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.13977},
}
Figure