TMPNN: High-Order Polynomial Regression Based on Taylor Map Factorization

Andrei Ivanov, Stefan Maria AiluroAAAI 2024 · DOI

TL;DR

This paper introduces TMPNN, a novel high-order polynomial regression model that uses Taylor map factorization to efficiently capture complex relationships in data without an explosion of parameters. By framing regression as a discrete dynamical system, TMPNN offers a theoretically grounded, interpretable approach that performs competitively with state-of-the-art methods, particularly in multi-target and extrapolation scenarios.

What Problem Does This Paper Address?

Traditional regression methods often struggle with high-order polynomial relationships, leading to either an unmanageable number of parameters (curse of dimensionality) or poor extrapolation capabilities. Machine learning models like neural networks can capture complex patterns but often act as "black boxes," lacking interpretability. This paper addresses the challenge of building high-order polynomial regression models that are both computationally efficient and interpretable, especially for multi-target regression problems where multiple outputs depend on the same inputs.

Methods

The core innovation is the Taylor Map Polynomial Neural Network (TMPNN). Instead of directly fitting a high-order polynomial, TMPNN uses a Taylor map – a transformation that iteratively applies a lower-to-moderate order polynomial function. This is implemented as a neural network with shared weights across layers.

Here's a breakdown:

  • Taylor Map: A function $Z_{t+1} = M(Z_t) = W_0 + W_1 Z_t + \dots + W_k Z_t^{[k]}$ where $Z_t$ is a state vector, $W_i$ are trainable weights, and $Z_t^{[k]}$ is the $k$-th Kronecker power. This effectively represents a $k$-th order polynomial.
  • Iterative Application: The Taylor map is applied $p$ times, starting with an extended input vector $Z_0$ (original features plus $m$ zeros for $m$ targets). This process, $Z_p = M^p(Z_0)$, results in a high-order polynomial of degree $k^p$ in the original input features.
  • Shared Weights: Crucially, the weights $W_i$ are shared across all $p$ layers. This significantly reduces the number of trainable parameters compared to a direct high-order polynomial fit.
  • Connection to ODEs: The authors show that TMPNN can be interpreted as a numerical approximation of a system of ordinary differential equations (ODEs). This connection provides a theoretical foundation for its interpretability and extrapolation capabilities.

Datasets

The paper evaluates TMPNN on a variety of datasets:

  • Feynman Symbolic Regression Benchmark: 120 datasets derived from physics equations, used to test accuracy under varying noise levels.
  • UCI Regression Datasets: 32 datasets with fewer than 50 features, used for general regression performance comparison.
  • Friedman-1 Dataset: A synthetic dataset used to analyze the effect of noise, number of samples, and unimportant features.
  • Gas and Petrochemical Processing Dataset: A real-life industrial dataset from the Sibur2023 Challenge, used to demonstrate out-of-distribution prediction capabilities.
  • UCI Yacht Hydrodynamics Dataset: Used specifically to evaluate extrapolation performance.

Key Insights and Ideas

  • Efficient High-Order Polynomials: The Taylor map factorization with shared weights is a clever way to achieve very high-order polynomial regression without the typical parameter explosion. This is a significant advancement for modeling complex, non-linear relationships.
  • Interpretability via ODEs: The theoretical link to ODEs is a major contribution. It suggests that TMPNN models are not just black boxes but can be understood as approximating underlying dynamical systems, which is crucial for scientific and engineering applications.
  • Natural Multi-Target Regression: By extending the input vector with zeros for target variables, TMPNN inherently handles multi-target regression without needing separate models for each target, potentially capturing inter-target relationships.
  • Extrapolation Advantage: The ODE interpretation hints at why TMPNN might perform better at extrapolation. By modeling the system's dynamics, it can potentially predict behavior beyond the training data range more robustly than models that simply interpolate.
  • Weight Initialization for Deeper Models: The paper proposes a method (equation 15) to initialize weights for deeper TMPNNs (higher $p$) based on shallower ones, which helps maintain performance and avoid issues with optimization.

Key Results

  • Competitive Performance: TMPNN performs comparably to state-of-the-art methods like CatBoost and DNNR on various UCI datasets and the Feynman benchmark.
  • Outperforms on Specific Tasks:
    • It shows strong performance on the Sibur2023 gas processing dataset, winning first prize, particularly excelling in out-of-distribution prediction.
  • It demonstrates superior extrapolation capabilities on the UCI Yacht Hydrodynamics dataset compared to regular polynomial regression, TabNet, and other ML models.
  • Robustness to Noise: TMPNN shows good performance across different noise levels on the Feynman benchmark, with accuracy often outperforming other models.
  • Efficiency: TMPNN models are noted for their small serialized size (e.g., 0.2 MB for the Sibur2023 challenge solution), making them practical for deployment.

Limitations and Future Work

  • Hyperparameter Tuning: While the paper uses default hyperparameters for TMPNN in some experiments, optimal selection of $k$ (map order) and $p$ (number of steps) is crucial and requires further investigation.
  • Initialization Sensitivity: The paper notes that initializations other than identity mapping can lead to worse results, and increasing $p$ without proper initialization (using relation 15) can decrease performance, suggesting a need for robust initialization strategies.
  • Theoretical Guarantees: While the connection to ODEs is established, the paper acknowledges that theoretical aspects like convergence and error estimation with respect to $k$ and $p$ require further research.
  • Data Properties: The paper suggests future work to investigate the influence of data normalization, categorical/ordinal features, and the differentiability of target-feature relationships on TMPNN's performance.
  • Classification Extension: The authors propose extending TMPNN for classification problems by constraining trajectories to converge to target states, building on prior work on PNNs for classification.
  • Feature Engineering: Exploring alternative ways to initialize the state vector $Z_0$ (e.g., using learnable parameters or predictions from other models) or extending the state vector with latent units are suggested for improving model expressivity.

Summary updated

BibTeX
@inproceedings{ivanov2024tmpnn,
  author = {Andrei Ivanov and Stefan Maria Ailuro},
  title = {{TMPNN: High-Order Polynomial Regression Based on Taylor Map Factorization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29168},
}
Figure