Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks

Yihang Gao, Michael K. Ng, Vincent Y. F. Tan

TL;DR

Kolmogorov–Arnold Networks (KANs) are a promising alternative to traditional Multi-Layer Perceptrons (MLPs) for scientific tasks, but fine-tuning them for new tasks is computationally expensive. This paper introduces LoTRA (Low Tensor-Rank Adaptation), a parameter-efficient fine-tuning method that leverages Tucker tensor decomposition to update KANs at a fraction of the storage and compute cost. Applied to physics-informed machine learning, LoTRA successfully solves families of partial differential equations (PDEs) with minimal parameter overhead while maintaining high accuracy.

What Problem Does This Paper Address?

Kolmogorov–Arnold Networks (KANs) have recently taken the machine learning world by storm, particularly in scientific computing and physics-informed machine learning. Unlike standard MLPs, which place fixed activation functions on nodes and learn scalar weights, KANs place learnable univariate activation functions (typically parameterized by splines or polynomials) on the network's edges.

While KANs excel at function approximation and solving partial differential equations (PDEs), they face a major hurdle in practical deployment: transfer learning. In real-world engineering, researchers rarely solve just one PDE; they solve a class of PDEs with varying physical parameters (e.g., changing fluid viscosity or thermal conductivity). Training a full KAN from scratch for every parameter variation is computationally prohibitive, and storing a massive, separate network for every single scenario creates a storage nightmare.

Standard parameter-efficient fine-tuning (PEFT) methods like LoRA cannot be directly applied here. Because KAN layers map inputs to outputs across multiple basis functions, their parameters are structured as third-order tensors (Input Dimension $\times$ Output Dimension $\times$ Number of Basis Functions) rather than standard 2D weight matrices.

Methods

To bridge this gap, the authors propose LoTRA (Low Tensor-Rank Adaptation). Instead of updating the massive full parameter tensor $\mathcal{A}$, LoTRA freezes the pre-trained KAN and injects a low-rank update via Tucker decomposition—the tensor equivalent of Singular Value Decomposition (SVD).

The fine-tuned tensor update is formulated as: $$\mathcal{A}_{ft} = \mathcal{A}_{pt} + \mathcal{G} \times_1 \mathbf{U}^{(1)} \times_2 \mathbf{U}^{(2)} \times_3 \mathbf{U}^{(3)}$$

Where:

  • $\mathcal{G}$ is a significantly smaller learnable core tensor.
  • $\mathbf{U}^{(1)}, \mathbf{U}^{(2)}, \mathbf{U}^{(3)}$ are low-rank transformation matrices for each tensor mode.
Fig. 1. Singular values for the pre-trained model parameters and fine-tuned updates, demonstrating the low Tucker rank structure.

The Adaptive Learning Rate Strategy

A key theoretical contribution of this paper is the discovery that assigning identical learning rates to all LoTRA components leads to highly inefficient training. Through a rigorous first-order linearization analysis, the authors prove that the gradient updates for the transformation matrices scale with the input dimensions ($n$) and the number of basis functions ($nd$). To achieve stable and efficient training, they derive an mathematically grounded adaptive learning rate strategy:

  • Core tensor ($\mathcal{G}$) and output matrix ($\mathbf{U}^{(2)}$): $\eta = \Theta(1)$
  • Input matrix ($\mathbf{U}^{(1)}$): $\eta \propto 1/n$
  • Basis function matrix ($\mathbf{U}^{(3)}$): $\eta \propto 1/nd$

Slim KANs

The authors also introduce Slim KANs, which apply this low-tensor-rank structure from day one (initializing the base tensor to zero). This creates a permanently compressed KAN architecture that inherently mitigates overfitting through low-rank regularization.

Datasets

The authors evaluated LoTRA and Slim KANs across two primary domains:

  1. Physics-Informed PDE Benchmarks: Synthetic datasets representing three fundamental classes of second-order PDEs with varying physical parameters ($\lambda$):
  2. Elliptic equations (steady-state systems)
    • Allen-Cahn equations (nonlinear parabolic equations describing phase separation)
    • Hyperbolic equations (wave propagation models)
  3. Standard Image Classification: MNIST and CIFAR-10 (the latter integrated with a 1-layer ResNet) to test the generalization of Slim KANs outside of pure scientific computing.

Key Insights and Ideas

  • Intrinsic Low-Rank Structure: The authors empirically verify that both pre-trained KAN parameters and their fine-tuning updates naturally exhibit a low Tucker rank structure (as shown by the rapid decay of singular values in Figure 1). This justifies why tensor compression works so well without destroying network expressiveness.
  • The Multi-LR Necessity: In matrix-based LoRA, a single learning rate usually suffices. In tensor-based LoTRA, the mathematical interplay between modes means that ignoring the dimensionality of the input and basis functions causes gradient explosion or stagnation.
  • Implicit Regularization: When training Slim KANs on noisy data, the authors observed a distinct U-shaped validation curve relative to the compression ratio. Tightening the low-rank constraint effectively filters out high-frequency noise, acting as an excellent regularizer.

Key Results

  • PDE Accuracy: LoTRA consistently matched or outperformed full parameter fine-tuning ("full") and training from scratch ("zero"), while drastically outperforming standard MLP baselines.

For instance, look at the relative error percentages for elliptic equations under different parameter scales:

Relative error for elliptic equations (parameter = 0.1) Relative error for elliptic equations (parameter = 1.0)

  • Massive Storage Savings: With a compression ratio ($cr$) of $1/4$, each mode of the tensor is quartered, resulting in a parameter footprint for the adapted layers that is $(1/4)^3 = 1/64$ (less than 2%) of the original size.
  • Vision Tasks: While KANs historically lag slightly behind MLPs in pure computer vision tasks, Slim KANs combined with a ResNet on CIFAR-10 achieved up to 76.5% accuracy, outperforming comparable MLP-Small baselines (73.29%) while utilizing significantly fewer parameters.

Limitations and Future Work

  • Scope of Tasks: The primary focus of this work is heavily weighted toward physics-informed machine learning (PDEs). The effectiveness of LoTRA on large-scale computer vision or natural language processing tasks remains an open question.
  • Theoretical Simplifications: The optimal learning rate proofs were derived using a simplified one-layer toy case before being generalized to multi-layer networks via corollaries. A deeper, holistic theoretical analysis of multi-layer tensor interactions during gradient descent is a clear next step.

Summary updated

Abstract

Kolmogorov--Arnold networks (KANs) have demonstrated their potential as an alternative to multi-layer perceptions (MLPs) in various domains, especially for science-related tasks. However, transfer learning of KANs remains a relatively unexplored area. In this paper, inspired by Tucker decomposition of tensors and evidence on the low tensor-rank structure in KAN parameter updates, we develop low tensor-rank adaptation (LoTRA) for fine-tuning KANs. We study the expressiveness of LoTRA based on Tucker decomposition approximations. Furthermore, we provide a theoretical analysis to select the learning rates for each LoTRA component to enable efficient training. Our analysis also shows that using identical learning rates across all components leads to inefficient training, highlighting the need for an adaptive learning rate strategy. Beyond theoretical insights, we explore the application of LoTRA for efficiently solving various partial differential equations (PDEs) by fine-tuning KANs. Additionally, we propose Slim KANs that incorporate the inherent low-tensor-rank properties of KAN parameter tensors to reduce model size while maintaining superior performance. Experimental results validate the efficacy of the proposed learning rate selection strategy and demonstrate the effectiveness of LoTRA for transfer learning of KANs in solving PDEs. Further evaluations on Slim KANs for function representation and image classification tasks highlight the expressiveness of LoTRA and the potential for parameter reduction through low tensor-rank decomposition.
BibTeX
@article{gao2025low,
  author = {Yihang Gao and Michael K. Ng and Vincent Y. F. Tan},
  title = {{Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks}},
  journal = {IEEE Trans. Signal Process.},
  year = {2025},
}
Figure