Partition of Unity Networks: Deep HP-Approximation

Kookjin Lee, Nathaniel Trask, Ravi G. Patel, Mamikon A. Gulian, Eric C. Cyr

Abstract

Researchers from Sandia National Laboratories have introduced Partition of Unity Networks (POUnets), a hybrid architecture that combines the flexibility of deep learning with the rigorous accuracy of classical numerical analysis. By forcing a neural network to focus solely on partitioning space while using optimal polynomial fits for local approximation, POUnets achieve high-order convergence and outperform standard deep networks on complex, discontinuous functions.

Introduction

In the world of function approximation, a significant gap exists between Deep Neural Networks (DNNs) and Finite Element Methods (FEM). DNNs excel at "breaking the curse of dimensionality," identifying patterns in high-dimensional data where classical methods fail. However, they are notoriously difficult to train to high precision; gradient descent often plateaus long before reaching theoretical accuracy limits. Conversely, FEM and $hp$-approximation (where $h$ is mesh size and $p$ is polynomial degree) offer mathematically guaranteed, high-order accuracy but struggle with high-dimensional "meshes" and complex geometries.

The authors identify that while deep networks can theoretically emulate high-order numerical methods, they rarely do so in practice because optimizers struggle to simultaneously learn localization (where to look) and approximation (what the function looks like). POUnets resolve this by explicitly building a "Partition of Unity" into the network architecture.

Methods: Architecture and Optimization

POUnets represent a departure from black-box models by utilizing a sum of local experts:

  1. The Architecture: A DNN (either an RBF-Net or a ResNet) outputs a set of weight functions $\phi_\alpha(x)$ that sum to one (the Partition of Unity). These functions partition the input space into distinct regions.
  2. Local Polynomials: Inside each region, the function is approximated by a polynomial space with learnable coefficients.
  3. LSGD Optimizer: The authors employ a "Least-Squares Gradient Descent" (LSGD) strategy. In each training step, the partition is held fixed while a globally optimal least-squares problem is solved to find the best polynomial coefficients. A gradient descent step then refines the partition boundaries.
  4. Two-Phase Training: To prevent "partition collapse"—where one region dominates the entire dataset—a regularized pre-training phase encourages the network to create a "quasi-uniform" mesh before fine-tuning for accuracy.

Handling Discontinuities and High Frequencies

The model was evaluated on synthetic benchmarks designed to stress-test numerical convergence, including piecewise linear (triangular) and piecewise quadratic waves with increasing frequencies. Because the ResNet-based POU can learn sharp boundaries, it handles functions with many discontinuities far better than smooth global approximators like standard ResNets.

Figure 3: Comparison of relative error between ResNet and POUnet on triangular and quadratic waves across varying frequencies.

As shown in the figure above, standard ResNets failed to produce accurate approximations for high-frequency waves, often staying above 1% error. In contrast, POUnets accurately captured sharp peaks and valleys with sub-1% error, demonstrating significant robustness to complexity.

Convergence and Accuracy

A key insight of this work is the "separation of concerns." By delegating the fitting to a least-squares solver, the neural network is freed to perform classification—essentially determining which data points belong to which local polynomial expert. This allows the model to demonstrate "spectral-like" $hp$-convergence.

Figure 6: Convergence comparison between POUnets (varying polynomial degree p) and standard MLPs (varying depth and width).

On smooth analytic functions, POUnets achieved relative $\ell_2$ errors as low as $10^{-10}$, whereas standard MLPs stagnated around $10^{-2}$. As the number of partitions ($N_{part}$) or the polynomial degree ($p$) increases, the error in POUnets drops exponentially faster than in standard architectures.

Discussion and Future Work

While POUnets represent a breakthrough in high-precision deep learning, they are not without limitations. The two-phase training requires careful hyperparameter tuning, and the strong regularization needed in Phase 1 can lead to long training times. Furthermore, the current error analysis assumes highly localized partitions; future work will explore the performance of "fuzzier" partitions.

The ultimate goal for this architecture is its application to solving high-dimensional Partial Differential Equations (PDEs), where the combination of learned mesh-free partitions and high-order numerical accuracy could provide a significant advantage over existing solvers.

Summary updated

Abstract

Approximation theorists have established best-in-class optimal approximation rates of deep neural networks by utilizing their ability to simultaneously emulate partitions of unity and monomials. Motivated by this, we propose partition of unity networks (POUnets) which incorporate these elements directly into the architecture. Classification architectures of the type used to learn probability measures are used to build a meshfree partition of space, while polynomial spaces with learnable coefficients are associated to each partition. The resulting hp-element-like approximation allows use of a fast least-squares optimizer, and the resulting architecture size need not scale exponentially with spatial dimension, breaking the curse of dimensionality. An abstract approximation result establishes desirable properties to guide network design. Numerical results for two choices of architecture demonstrate that POUnets yield hp-convergence for smooth functions and consistently outperform MLPs for piecewise polynomial functions with large numbers of discontinuities.
BibTeX
@inproceedings{lee2021partition,
  author = {Kookjin Lee and Nathaniel Trask and Ravi G. Patel and Mamikon A. Gulian and Eric C. Cyr},
  title = {{Partition of Unity Networks: Deep HP-Approximation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=hrTKbNy13TT},
}
Figure