Abstract
Researchers from Sandia National Laboratories have introduced Partition of Unity Networks (POUnets), a hybrid architecture that combines the flexibility of deep learning with the rigorous accuracy of classical numerical analysis. By forcing a neural network to focus solely on partitioning space while using optimal polynomial fits for local approximation, POUnets achieve high-order convergence and outperform standard deep networks on complex, discontinuous functions.
Introduction
In the world of function approximation, a significant gap exists between Deep Neural Networks (DNNs) and Finite Element Methods (FEM). DNNs excel at "breaking the curse of dimensionality," identifying patterns in high-dimensional data where classical methods fail. However, they are notoriously difficult to train to high precision; gradient descent often plateaus long before reaching theoretical accuracy limits. Conversely, FEM and $hp$-approximation (where $h$ is mesh size and $p$ is polynomial degree) offer mathematically guaranteed, high-order accuracy but struggle with high-dimensional "meshes" and complex geometries.
The authors identify that while deep networks can theoretically emulate high-order numerical methods, they rarely do so in practice because optimizers struggle to simultaneously learn localization (where to look) and approximation (what the function looks like). POUnets resolve this by explicitly building a "Partition of Unity" into the network architecture.
Methods: Architecture and Optimization
POUnets represent a departure from black-box models by utilizing a sum of local experts:
- The Architecture: A DNN (either an RBF-Net or a ResNet) outputs a set of weight functions $\phi_\alpha(x)$ that sum to one (the Partition of Unity). These functions partition the input space into distinct regions.
- Local Polynomials: Inside each region, the function is approximated by a polynomial space with learnable coefficients.
- LSGD Optimizer: The authors employ a "Least-Squares Gradient Descent" (LSGD) strategy. In each training step, the partition is held fixed while a globally optimal least-squares problem is solved to find the best polynomial coefficients. A gradient descent step then refines the partition boundaries.
- Two-Phase Training: To prevent "partition collapse"—where one region dominates the entire dataset—a regularized pre-training phase encourages the network to create a "quasi-uniform" mesh before fine-tuning for accuracy.
Handling Discontinuities and High Frequencies
The model was evaluated on synthetic benchmarks designed to stress-test numerical convergence, including piecewise linear (triangular) and piecewise quadratic waves with increasing frequencies. Because the ResNet-based POU can learn sharp boundaries, it handles functions with many discontinuities far better than smooth global approximators like standard ResNets.
As shown in the figure above, standard ResNets failed to produce accurate approximations for high-frequency waves, often staying above 1% error. In contrast, POUnets accurately captured sharp peaks and valleys with sub-1% error, demonstrating significant robustness to complexity.
Convergence and Accuracy
A key insight of this work is the "separation of concerns." By delegating the fitting to a least-squares solver, the neural network is freed to perform classification—essentially determining which data points belong to which local polynomial expert. This allows the model to demonstrate "spectral-like" $hp$-convergence.
On smooth analytic functions, POUnets achieved relative $\ell_2$ errors as low as $10^{-10}$, whereas standard MLPs stagnated around $10^{-2}$. As the number of partitions ($N_{part}$) or the polynomial degree ($p$) increases, the error in POUnets drops exponentially faster than in standard architectures.
Discussion and Future Work
While POUnets represent a breakthrough in high-precision deep learning, they are not without limitations. The two-phase training requires careful hyperparameter tuning, and the strong regularization needed in Phase 1 can lead to long training times. Furthermore, the current error analysis assumes highly localized partitions; future work will explore the performance of "fuzzier" partitions.
The ultimate goal for this architecture is its application to solving high-dimensional Partial Differential Equations (PDEs), where the combination of learned mesh-free partitions and high-order numerical accuracy could provide a significant advantage over existing solvers.
Summary updated