Abstract
Physics-Informed Neural Networks (PINNs) have emerged as a powerful tool for solving partial differential equations (PDEs) by embedding physical laws into the neural network's loss function. However, they are frequently hindered by "spectral bias," where the model prioritizes low-frequency, smooth components at the expense of high-frequency oscillations and sharp transitions. We introduce xLSTM-PINN, which addresses this by integrating memory-gated xLSTM blocks into the PINN architecture. By leveraging gated memory mechanisms and residual micro-steps, xLSTM-PINN enhances spectral sensitivity and representational power. Our evaluation across multiple PDE benchmarks shows that xLSTM-PINN achieves up to a 4,000x improvement in accuracy over baseline models, providing a robust architectural solution to the spectral bias problem.
Introduction: Escaping the "Spectral-Bias Dungeon"
Standard PINNs typically utilize Multi-Layer Perceptrons (MLPs), which are known to be "lazy" learners. In complex physical simulations—such as fluid turbulence or heat transfer with sharp boundaries—MLPs often produce blurred results because they struggle to capture high-frequency details. This phenomenon, termed spectral bias, limits the applicability of PINNs in high-fidelity scientific computing.
While previous attempts to mitigate this have involved complex sampling or Fourier features, these often require extensive manual tuning. xLSTM-PINN proposes an architectural-level fix that "lifts" the tail of the kernel spectrum, making the network inherently more sensitive to high-frequency data.
Methodology: xLSTM-PINN Architecture
The core innovation of xLSTM-PINN is the replacement of standard hidden layers with xLSTM blocks. This architecture introduces two primary advantages:
- Gated Memory Mechanisms: These allow the network to selectively retain or discard information, enabling the precise modeling of features that standard MLPs might overlook.
- Residual Micro-steps: The model performs multiple internal iterations within a single layer using shared parameters. This increases the "effective depth" of the computation without significantly increasing the number of trainable weights.
Crucially, xLSTM-PINN maintains the standard physics loss formulation and Automatic Differentiation (AD) process. The architecture also incorporates a stabilization scaling method to ensure numerical robustness during deep physics-informed learning. Theoretical analysis using Neural Tangent Kernel (NTK) theory confirms that this specific architecture shifts the resolvable bandwidth, allowing the network to learn much finer details than a standard MLP.
Experimental Evaluation
The model was rigorously tested against four diverse PDE benchmarks:
- 2D Laplace Equation: Featuring mixed Dirichlet-Neumann boundary conditions.
- Steady-State Heat Conduction: Modeling convective (Robin) boundaries on a circular silicon plate.
- Anisotropic Poisson-Beam Equation: A challenging 4th-order PDE testing high-order derivative handling.
- 1D Advection-Reaction: Simulating transport phenomena through a "drift-decay" model.
Key Results and Insights
Across all benchmarks, xLSTM-PINN demonstrated superior performance:
- Unprecedented Accuracy: In the 2D Laplace case, the Mean Squared Error (MSE) dropped from $6.04 \times 10^{-5}$ (baseline) to $1.47 \times 10^{-8}$, representing a nearly 4,000x improvement.
- High-Order PDE Robustness: For the 4th-order Poisson-Beam equation, xLSTM-PINN achieved an MSE of $1.93 \times 10^{-6}$, significantly outperforming the baseline ($5.94 \times 10^{-5}$).
- Enhanced Convergence: Loss history analysis indicates that xLSTM-PINN reaches lower error floors more rapidly and with greater stability than standard MLPs.
- Visual Fidelity: Absolute error maps confirm that xLSTM-PINN effectively eliminates the "warm" error bands typical of standard PINNs, reducing error to near-numerical noise levels.
Conclusion and Future Directions
xLSTM-PINN represents a significant step forward in physics-informed machine learning by providing an architectural solution to spectral bias. While the "micro-steps" introduce some computational overhead per iteration, the gains in accuracy and parameter efficiency are substantial. Future work will focus on scaling this architecture to high-dimensional, time-dependent systems such as 3D turbulent fluid simulations and chaotic Navier-Stokes equations.
Summary updated