Low Tensor-Rank Adaptation of Kolmogorov-Arnold Networks

Yihang Gao, Michael K. P. Ng, Vincent Y. F. Tan

Abstract

Kolmogorov-Arnold Networks (KANs) have emerged as a powerful alternative to Multi-Layer Perceptrons (MLPs), particularly for scientific computing and solving partial differential equations (PDEs). However, KANs face challenges regarding parameter efficiency and the computational cost of transfer learning. We introduce LoTRA (Low Tensor-Rank Adaptation), a parameter-efficient fine-tuning method that utilizes Tucker decomposition to exploit the inherent low-rank structure of KAN parameter updates. Our approach enables efficient transfer learning for physics-informed tasks and introduces "Slim KANs" that maintain high performance with significantly fewer parameters. Furthermore, we derive a theoretically grounded heterogeneous learning rate strategy to optimize training efficiency across Tucker components.

Introduction

While KANs offer superior performance in representing complex functions compared to standard MLPs, their 3rd-order tensor weight structure (input dimension × output dimension × number of basis functions) leads to high parameter counts. This makes fine-tuning for specific tasks, such as solving a class of PDEs with varying physical parameters, computationally expensive. Traditional Low-Rank Adaptation (LoRA) techniques designed for matrix-based MLPs do not directly apply to this tensor structure. LoTRA addresses this gap by applying Tucker decomposition to the parameter updates, significantly reducing the number of trainable parameters while preserving model expressivity.

Methodology

The LoTRA Framework

LoTRA keeps pre-trained weights $\mathcal{A}_{pt}$ frozen and introduces a low-rank update through Tucker decomposition: $$\mathcal{A}_{ft} = \mathcal{A}_{pt} + \mathcal{G} \times_1 U^{(1)} \times_2 U^{(2)} \times_3 U^{(3)}$$ where $\mathcal{G}$ is a small core tensor and $U^{(i)}$ are transformation matrices. This structure allows for a drastic reduction in the number of parameters updated during fine-tuning. By initializing the frozen component to zero, we create Slim KANs, which are inherently compressed architectures suitable for resource-constrained environments.

Heterogeneous Learning Rate Selection

A key contribution of this work is the derivation of a theoretically grounded learning rate strategy. We demonstrate that uniform learning rates across all Tucker components lead to training instability or slow convergence as models scale. Our analysis suggests a heterogeneous strategy:

  • Core tensor ($\mathcal{G}$) and Output-mode matrix ($U^{(2)}$): Standard learning rate $\Theta(1)$.
  • Input-mode matrix ($U^{(1)}$) and Basis-mode matrix ($U^{(3)}$): Scaled by the inverse of the layer width and number of basis functions, respectively ($\Theta(n^{-1})$).

Empirical Evidence and Results

Low-Rank Structure

The motivation for LoTRA is grounded in the observation that KAN parameter updates exhibit a distinct low-rank structure. Using Higher-Order Singular Value Decomposition (HOSVD), we observe a rapid decay in singular values across different indices, justifying the use of Tucker decomposition for parameter adaptation.

Singular value decay demonstrating the low-rank nature of KAN parameter updates.

Performance in Physics-Informed Tasks

We evaluated LoTRA across various second-order PDEs, including Elliptic, Allen-Cahn, and Hyperbolic equations. In transfer learning scenarios, LoTRA achieved relative errors comparable to full fine-tuning while using only a fraction of the parameters.

Performance comparison (relative error %) of LoTRA against full and zero-initialized KAN variants for Elliptic PDEs.

As shown in Table I, ChebyKAN-LoTRA with a compression ratio (cr) of 1/4 achieved a relative error of 0.06%, matching the performance of the full fine-tuned model. This demonstrates that the low-rank constraint can act as an effective regularizer, filtering noise and preventing overfitting during adaptation on limited data.

Computer Vision Benchmarks

Beyond scientific computing, Slim KANs were tested on MNIST and CIFAR-10. On MNIST, Slim KANs achieved over 97% accuracy with significantly reduced parameter counts. When integrated into a ResNet backbone for CIFAR-10, Slim KANs outperformed baseline MLP models of comparable size, indicating their versatility across domains.

Conclusion

LoTRA provides a robust framework for making Kolmogorov-Arnold Networks more practical for real-world applications. By leveraging tensor decomposition and theoretically derived learning rates, we enable efficient adaptation and compression without sacrificing the high accuracy that makes KANs attractive for scientific machine learning. Future work will explore the scalability of LoTRA in massive-scale architectures like Vision Transformers and Large Language Models.

Summary updated

Abstract

Kolmogorov--Arnold networks (KANs) have demonstrated their potential as an alternative to multi-layer perceptions (MLPs) in various domains, especially for science-related tasks. However, transfer learning of KANs remains a relatively unexplored area. In this paper, inspired by Tucker decomposition of tensors and evidence on the low tensor-rank structure in KAN parameter updates, we develop low tensor-rank adaptation (LoTRA) for fine-tuning KANs. We study the expressiveness of LoTRA based on Tucker decomposition approximations. Furthermore, we provide a theoretical analysis to select the learning rates for each LoTRA component to enable efficient training. Our analysis also shows that using identical learning rates across all components leads to inefficient training, highlighting the need for an adaptive learning rate strategy. Beyond theoretical insights, we explore the application of LoTRA for efficiently solving various partial differential equations (PDEs) by fine-tuning KANs. Additionally, we propose Slim KANs that incorporate the inherent low-tensor-rank properties of KAN parameter tensors to reduce model size while maintaining superior performance. Experimental results validate the efficacy of the proposed learning rate selection strategy and demonstrate the effectiveness of LoTRA for transfer learning of KANs in solving PDEs. Further evaluations on Slim KANs for function representation and image classification tasks highlight the expressiveness of LoTRA and the potential for parameter reduction through low tensor-rank decomposition.
BibTeX
@article{gao2025low,
  author = {Yihang Gao and Michael K. P. Ng and Vincent Y. F. Tan},
  title = {{Low Tensor-Rank Adaptation of Kolmogorov-Arnold Networks}},
  journal = {IEEE Trans. Signal Process.},
  year = {2025},
}
Figure