Abstract
Kolmogorov-Arnold Networks (KANs) have emerged as a powerful alternative to Multi-Layer Perceptrons (MLPs), particularly for scientific computing and solving partial differential equations (PDEs). However, KANs face challenges regarding parameter efficiency and the computational cost of transfer learning. We introduce LoTRA (Low Tensor-Rank Adaptation), a parameter-efficient fine-tuning method that utilizes Tucker decomposition to exploit the inherent low-rank structure of KAN parameter updates. Our approach enables efficient transfer learning for physics-informed tasks and introduces "Slim KANs" that maintain high performance with significantly fewer parameters. Furthermore, we derive a theoretically grounded heterogeneous learning rate strategy to optimize training efficiency across Tucker components.
Introduction
While KANs offer superior performance in representing complex functions compared to standard MLPs, their 3rd-order tensor weight structure (input dimension × output dimension × number of basis functions) leads to high parameter counts. This makes fine-tuning for specific tasks, such as solving a class of PDEs with varying physical parameters, computationally expensive. Traditional Low-Rank Adaptation (LoRA) techniques designed for matrix-based MLPs do not directly apply to this tensor structure. LoTRA addresses this gap by applying Tucker decomposition to the parameter updates, significantly reducing the number of trainable parameters while preserving model expressivity.
Methodology
The LoTRA Framework
LoTRA keeps pre-trained weights $\mathcal{A}_{pt}$ frozen and introduces a low-rank update through Tucker decomposition: $$\mathcal{A}_{ft} = \mathcal{A}_{pt} + \mathcal{G} \times_1 U^{(1)} \times_2 U^{(2)} \times_3 U^{(3)}$$ where $\mathcal{G}$ is a small core tensor and $U^{(i)}$ are transformation matrices. This structure allows for a drastic reduction in the number of parameters updated during fine-tuning. By initializing the frozen component to zero, we create Slim KANs, which are inherently compressed architectures suitable for resource-constrained environments.
Heterogeneous Learning Rate Selection
A key contribution of this work is the derivation of a theoretically grounded learning rate strategy. We demonstrate that uniform learning rates across all Tucker components lead to training instability or slow convergence as models scale. Our analysis suggests a heterogeneous strategy:
- Core tensor ($\mathcal{G}$) and Output-mode matrix ($U^{(2)}$): Standard learning rate $\Theta(1)$.
- Input-mode matrix ($U^{(1)}$) and Basis-mode matrix ($U^{(3)}$): Scaled by the inverse of the layer width and number of basis functions, respectively ($\Theta(n^{-1})$).
Empirical Evidence and Results
Low-Rank Structure
The motivation for LoTRA is grounded in the observation that KAN parameter updates exhibit a distinct low-rank structure. Using Higher-Order Singular Value Decomposition (HOSVD), we observe a rapid decay in singular values across different indices, justifying the use of Tucker decomposition for parameter adaptation.
Performance in Physics-Informed Tasks
We evaluated LoTRA across various second-order PDEs, including Elliptic, Allen-Cahn, and Hyperbolic equations. In transfer learning scenarios, LoTRA achieved relative errors comparable to full fine-tuning while using only a fraction of the parameters.
As shown in Table I, ChebyKAN-LoTRA with a compression ratio (cr) of 1/4 achieved a relative error of 0.06%, matching the performance of the full fine-tuned model. This demonstrates that the low-rank constraint can act as an effective regularizer, filtering noise and preventing overfitting during adaptation on limited data.
Computer Vision Benchmarks
Beyond scientific computing, Slim KANs were tested on MNIST and CIFAR-10. On MNIST, Slim KANs achieved over 97% accuracy with significantly reduced parameter counts. When integrated into a ResNet backbone for CIFAR-10, Slim KANs outperformed baseline MLP models of comparable size, indicating their versatility across domains.
Conclusion
LoTRA provides a robust framework for making Kolmogorov-Arnold Networks more practical for real-world applications. By leveraging tensor decomposition and theoretically derived learning rates, we enable efficient adaptation and compression without sacrificing the high accuracy that makes KANs attractive for scientific machine learning. Future work will explore the scalability of LoTRA in massive-scale architectures like Vision Transformers and Large Language Models.
Summary updated