Experimental observation on a low-rank tensor model for eigenvalue problems

Jun Hu, Pengzhan Jin

TL;DR

Researchers from Peking University have demonstrated that for high-dimensional eigenvalue problems, a simple Polynomial-based Low-rank Tensor Model (PLTM) outperforms complex Tensor Neural Networks (TNNs). By replacing neural networks with analytic polynomial bases, the authors achieved near-machine precision on problems with up to 512 dimensions, primarily because the model allows for exact integration rather than numerical approximation.

What Problem Does This Paper Address?

Solving partial differential equations (PDEs) and eigenvalue problems in high dimensions (e.g., $d=512$) is notoriously difficult due to the "curse of dimensionality." Traditional grid-based methods fail because the number of points required grows exponentially with the dimension.

Recent trends have leaned toward neural network-based solvers (like PINNs or the Deep Ritz method). While effective, these often rely on Monte-Carlo integration or quadrature schemes to handle the high-dimensional integrals required by the physics. These integration methods introduce errors and computational overhead. The authors ask a fundamental question: Is the "neural network" part of a Tensor Neural Network actually necessary, or is the "low-rank tensor" structure doing all the heavy lifting?

Methods

The authors compare two main architectures:

  1. Tensor Neural Network (TNN): Approximates a high-dimensional function as a sum of products of univariate neural networks.
  2. Low-rank Tensor Model (LTM): Similar to TNN, but uses fixed analytic basis functions instead of neural networks.
  3. PLTM: Specifically uses Legendre polynomials as the basis.

The core advantage of the PLTM is that the integral of a product of polynomials is simply the product of the integrals of those polynomials. This allows the researchers to calculate the loss function (the Rayleigh quotient for eigenvalue problems) exactly, eliminating the integration error that plagues neural network approaches.

They optimize the coefficients of these polynomials using the Adam optimizer, treating the problem as a machine learning task where the "weights" are the coefficients of the tensor decomposition.

Datasets

The model was evaluated on three benchmarks:

  • The Laplacian Operator: A fundamental physics problem on a hypercube ($d=10$ and $d=512$).
  • The Harmonic Oscillator: A quantum mechanics-style problem involving a potential term ($d=10$ and $d=512$).
  • MNIST: The classic handwritten digit dataset, used to test if this physics-oriented model generalizes to standard machine learning classification tasks.

Key Insights and Ideas

  • Exact Integration is King: In physics-informed machine learning, the error often comes from how we calculate the "physics" part (the integrals). By using polynomials, the authors moved from approximating the physics to calculating it exactly.
  • Simplicity Scales: The PLTM achieved better results than TNNs while being "simpler." It suggests that for problems with smooth solutions (like many physical systems), the universal approximation power of neural networks might be overkill compared to classical basis functions arranged in a smart, low-rank tensor format.
  • Initialization Matters: The authors found that a specific initialization strategy (setting the first basis coefficient to a constant and others to zero) was vital to prevent the high-dimensional products from exploding or vanishing during training.

Key Results

  • Superior Accuracy: For the Laplacian operator in 512 dimensions, the PLTM achieved a relative error of $1.2 \times 10^{-13}$ (near machine precision), whereas the TNN only reached $1.6 \times 10^{-7}$.
  • Efficiency: The training process for PLTM is significantly faster than TNN because it avoids the overhead of evaluating neural networks and performing numerical integration at every step.
  • MNIST Performance: On image classification, the PLTM performed respectably (~95% accuracy), matching the TNN and beating a linear model (~92%), though it fell short of a standard nonlinear Deep Neural Network (~98%). This suggests that while PLTM is a powerhouse for physics, the "jagged" nature of image data still benefits from the nonlinearities of traditional deep learning.

Limitations and Future Work

The authors acknowledge a few constraints:

  • Boundary Conditions: The current PLTM relies on specific linear combinations of polynomials to satisfy zero-boundary conditions. This might become more complex for irregular geometries.
  • Basis Selection: While Legendre polynomials worked for these cases, other problems might require different bases. The authors suggest Fourier bases for periodic problems as a next step.
  • Machine Error: As accuracy reaches $10^{-16}$, floating-point errors become a bottleneck, requiring careful management of numerical stability.

Summary updated

Abstract

Here we utilize a low-rank tensor model (LTM) as a function approximator, combined with the gradient descent method, to solve eigenvalue problems including the Laplacian operator and the harmonic oscillator. Experimental results show the superiority of the polynomial-based low-rank tensor model (PLTM) compared to the tensor neural network (TNN). We also test such low-rank architectures for the classification problem on the MNIST dataset.
BibTeX
@misc{hu2023experimental,
  author = {Jun Hu and Pengzhan Jin},
  title = {{Experimental observation on a low-rank tensor model for eigenvalue problems}},
  year = {2023},
  eprint = {2302.00538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.00538},
}
Figure