RPN: Reconciled Polynomial Network Towards Unifying PGMs, Kernel SVMs, MLP and KAN

Jiawei ZhangarXiv 2024 · DOI · arXiv

Abstract

The Reconciled Polynomial Network (RPN) is a novel deep learning framework that unifies diverse models—from Probabilistic Graphical Models and SVMs to MLPs and Kolmogorov-Arnold Networks (KAN)—into a single canonical representation. By leveraging Taylor’s Theorem to disentangle data expansion from parameter management, RPN achieves significantly higher accuracy and parameter efficiency across function fitting, vision, and language tasks.

Introduction

The field of artificial intelligence has evolved through distinct "eras," each defined by specific mathematical foundations and architectural paradigms. From the probabilistic reasoning of Probabilistic Graphical Models (PGMs) to the high-dimensional margin maximization of Support Vector Machines (SVMs), and finally to the current dominance of Deep Learning (MLPs, Transformers), these eras have largely remained fragmented.

Timeline of AI eras and the positioning of RPN as a unifying framework.

The RPN addresses this fragmentation by providing a unified mathematical framework. It aims to bridge the gap between interpretability and computational efficiency, specifically tackling the "curse of dimensionality" that often hinders high-order polynomial models.

The RPN Framework

The core of RPN is grounded in Taylor’s Theorem, which posits that any smooth function can be approximated by a sum of polynomials. RPN formalizes this into three distinct component functions:

  1. Data Expansion ($\\kappa$): This function projects input data into a high-dimensional space. RPN supports various expansion methods, including Taylor polynomials, B-splines, Fourier series, and probability density functions.
  2. Parameter Reconciliation ($\\psi$): To prevent the parameter explosion associated with high-dimensional expansion, RPN "fabricates" a large weight matrix from a smaller set of learnable parameters. This is achieved through techniques like low-rank decomposition or Kronecker products.
  3. Remainder Function ($\\pi$): Acting as a residual term (similar to skip-connections in ResNet), this function captures information that the polynomial approximation might otherwise miss.

By configuring these components, RPN can effectively "recreate" existing architectures like MLPs, KANs, or SVMs, establishing itself as a canonical representation for diverse model types.

Experimental Evaluation

The authors evaluated RPN across a wide range of benchmarks to demonstrate its versatility and performance.

Function Fitting

On the Feynman physics equations—a set of 100 real-world physics formulas—RPN significantly outperformed both MLPs and Kolmogorov-Arnold Networks (KANs). In many cases, RPN achieved Mean Squared Errors (MSE) that were 10 to 100 times lower than competing models.

Computer Vision and NLP

On the MNIST dataset, RPN reached 98.6% accuracy, surpassing KAN and MLP while maintaining high parameter efficiency. In Natural Language Processing tasks (IMDB, AGNews), RPN demonstrated superior robustness. While KAN models often struggle with sparse text data, RPN handled these inputs effectively, outperforming standard MLPs.

Tabular and Probabilistic Modeling

When utilizing probabilistic expansion functions, RPN outperformed traditional Bayesian and Markov networks on tabular datasets (e.g., Pima Indians Diabetes). This suggests that RPN can learn complex variable dependencies more effectively than models specifically designed for probabilistic graphical modeling.

Key Insights

  • Disentanglement: By separating data expansion from parameter management, RPN enables high-complexity modeling without prohibitive memory overhead.
  • Interpretability: RPN components offer concrete physical meanings. For instance, Taylor expansion visualizations in MNIST tasks show the model capturing high-order signals from central pixels to identify digits.
  • Biological Parallelism: The architecture draws parallels to biological neurons, where data expansion corresponds to complex spiking states and parameter reconciliation mimics the fabrication of neurotransmitters in synapses.

Conclusion and Future Work

RPN represents a significant step toward a unified theory of function learning. While currently highly effective, future work involves expanding model coverage to include tree-based models and unsupervised learning. Additionally, optimizing the data expansion step for high-performance computing (HPC) and exploring non-gradient-based training algorithms are key areas for development.

The authors have released the tinyBIG toolkit on GitHub to facilitate community experimentation with RPN architectures.

The tinyBIG toolkit logo.

Summary updated

BibTeX
@misc{zhang2024rpn,
  author = {Jiawei Zhang},
  title = {{RPN: Reconciled Polynomial Network Towards Unifying PGMs, Kernel SVMs, MLP and KAN}},
  year = {2024},
  eprint = {2407.04819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.04819},
}
Figure