Partition of unity networks: deep hp-approximation

Kookjin Lee, Nathaniel A. Trask, Ravi G. Patel, Mamikon A. Gulian, Eric C. Cyr

Abstract

Approximation theorists have established best-in-class optimal approximation rates of deep neural networks by utilizing their ability to simultaneously emulate partitions of unity and monomials. Motivated by this, we propose partition of unity networks (POUnets) which incorporate these elements directly into the architecture. Classification architectures of the type used to learn probability measures are used to build a meshfree partition of space, while polynomial spaces with learnable coefficients are associated to each partition. The resulting hp-element-like approximation allows use of a fast least-squares optimizer, and the resulting architecture size need not scale exponentially with spatial dimension, breaking the curse of dimensionality. Numerical results for two choices of architecture demonstrate that POUnets yield hp-convergence for smooth functions and consistently outperform MLPs for piecewise polynomial functions with large numbers of discontinuities.

Introduction

Deep neural networks (DNNs) are theoretically universal approximators, capable of representing complex functions with exponential accuracy. However, practical training often hits an optimization error barrier where standard architectures struggle to simultaneously localize regions of space and fit data within them. Classical numerical methods, such as the finite element method, utilize hp-approximation—refining mesh size (h) and increasing polynomial degree (p)—to achieve high precision. While effective, these methods suffer from the "curse of dimensionality" as mesh requirements grow exponentially with spatial dimensions. Partition of Unity Networks (POUnets) are introduced to combine the mesh-free flexibility of neural networks with the rigorous convergence properties of hp-elements.

The POUnet Architecture

The core of the POUnet is the mathematical structure of a Partition of Unity (POU), a set of non-negative functions {phi_alpha} that sum to unity across the domain. The architecture is split into two functional layers:

  1. The POU Layer: A neural network (e.g., a shallow RBF-Net or a deep ResNet) acts as a classifier to partition the input space into N_part regions. A softmax activation ensures the partition functions sum to one.
  2. The Polynomial Layer: Each partition is associated with a local polynomial space. The final output is a weighted sum: y_POU(x) = sum phi_alpha(x) * Polynomial_alpha(x).

By hard-coding the polynomial structure, the DNN can focus exclusively on its strength: adaptive partitioning and classification.

Optimization and Training

POUnets utilize a hybrid Least-Squares Gradient Descent (LSGD) optimizer. This block coordinate descent strategy alternates between:

  • Step A: Fixing POU parameters and solving a global linear least-squares problem for the optimal polynomial coefficients.
  • Step B: Updating the POU parameters (the "mesh") via a gradient descent step.

A two-phase training strategy is employed to ensure stability. Phase 1 applies L2 regularization to polynomial coefficients to prevent "partition collapse," ensuring a quasi-uniform distribution of the mesh before fine-tuning in Phase 2.

Experimental Evaluation

The performance of POUnets was benchmarked against standard MLPs and ResNets using synthetic datasets including smooth analytic functions and piecewise smooth functions (triangular and quadratic waves).

POUnets demonstrate algebraic convergence on smooth functions, reaching errors of 10^-10, while standard MLPs plateau near 10^-2.

For smooth functions, POUnets achieved "algebraic convergence," reaching relative l2-errors as low as 10^-10. In contrast, standard MLPs plateaued at approximately 10^-2. This demonstrates the network's ability to overcome the optimization barriers typical of standard DNNs.

POUnets consistently outperform ResNets on piecewise linear (triangular) and piecewise quadratic waves across varying frequencies.

On piecewise smooth functions with high-frequency oscillations and discontinuities, POUnets maintained sub-1% error. Standard ResNets significantly failed to capture these sharp features as the complexity of the waves increased.

Discussion and Conclusion

POUnets effectively break the curse of dimensionality; theoretical analysis shows that approximation error depends on the latent dimension of the data manifold rather than the ambient space dimension. This mesh-free adaptivity allows the network to "learn" an optimal mesh that wraps around the features of the target function. While hyperparameter tuning for regularization remains a challenge, the potential for POUnets to solve high-dimensional partial differential equations (PDEs) is a promising avenue for future research.

Summary updated

Abstract

Approximation theorists have established best-in-class optimal approximation rates of deep neural networks by utilizing their ability to simultaneously emulate partitions of unity and monomials. Motivated by this, we propose partition of unity networks (POUnets) which incorporate these elements directly into the architecture. Classification architectures of the type used to learn probability measures are used to build a meshfree partition of space, while polynomial spaces with learnable coefficients are associated to each partition. The resulting hp-element-like approximation allows use of a fast least-squares optimizer, and the resulting architecture size need not scale exponentially with spatial dimension, breaking the curse of dimensionality. An abstract approximation result establishes desirable properties to guide network design. Numerical results for two choices of architecture demonstrate that POUnets yield hp-convergence for smooth functions and consistently outperform MLPs for piecewise polynomial functions with large numbers of discontinuities.
BibTeX
@inproceedings{lee2021partition,
  author = {Kookjin Lee and Nathaniel A. Trask and Ravi G. Patel and Mamikon A. Gulian and Eric C. Cyr},
  title = {{Partition of unity networks: deep hp-approximation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=hrTKbNy13TT},
}
Figure