TL;DR
Kolmogorov–Arnold Networks (KANs) are a promising alternative to traditional Multi-Layer Perceptrons (MLPs) for scientific tasks, but fine-tuning them for new tasks is computationally expensive. This paper introduces LoTRA (Low Tensor-Rank Adaptation), a parameter-efficient fine-tuning method that leverages Tucker tensor decomposition to update KANs at a fraction of the storage and compute cost. Applied to physics-informed machine learning, LoTRA successfully solves families of partial differential equations (PDEs) with minimal parameter overhead while maintaining high accuracy.
What Problem Does This Paper Address?
Kolmogorov–Arnold Networks (KANs) have recently taken the machine learning world by storm, particularly in scientific computing and physics-informed machine learning. Unlike standard MLPs, which place fixed activation functions on nodes and learn scalar weights, KANs place learnable univariate activation functions (typically parameterized by splines or polynomials) on the network's edges.
While KANs excel at function approximation and solving partial differential equations (PDEs), they face a major hurdle in practical deployment: transfer learning. In real-world engineering, researchers rarely solve just one PDE; they solve a class of PDEs with varying physical parameters (e.g., changing fluid viscosity or thermal conductivity). Training a full KAN from scratch for every parameter variation is computationally prohibitive, and storing a massive, separate network for every single scenario creates a storage nightmare.
Standard parameter-efficient fine-tuning (PEFT) methods like LoRA cannot be directly applied here. Because KAN layers map inputs to outputs across multiple basis functions, their parameters are structured as third-order tensors (Input Dimension $\times$ Output Dimension $\times$ Number of Basis Functions) rather than standard 2D weight matrices.
Methods
To bridge this gap, the authors propose LoTRA (Low Tensor-Rank Adaptation). Instead of updating the massive full parameter tensor $\mathcal{A}$, LoTRA freezes the pre-trained KAN and injects a low-rank update via Tucker decomposition—the tensor equivalent of Singular Value Decomposition (SVD).
The fine-tuned tensor update is formulated as: $$\mathcal{A}_{ft} = \mathcal{A}_{pt} + \mathcal{G} \times_1 \mathbf{U}^{(1)} \times_2 \mathbf{U}^{(2)} \times_3 \mathbf{U}^{(3)}$$
Where:
- $\mathcal{G}$ is a significantly smaller learnable core tensor.
- $\mathbf{U}^{(1)}, \mathbf{U}^{(2)}, \mathbf{U}^{(3)}$ are low-rank transformation matrices for each tensor mode.
The Adaptive Learning Rate Strategy
A key theoretical contribution of this paper is the discovery that assigning identical learning rates to all LoTRA components leads to highly inefficient training. Through a rigorous first-order linearization analysis, the authors prove that the gradient updates for the transformation matrices scale with the input dimensions ($n$) and the number of basis functions ($nd$). To achieve stable and efficient training, they derive an mathematically grounded adaptive learning rate strategy:
- Core tensor ($\mathcal{G}$) and output matrix ($\mathbf{U}^{(2)}$): $\eta = \Theta(1)$
- Input matrix ($\mathbf{U}^{(1)}$): $\eta \propto 1/n$
- Basis function matrix ($\mathbf{U}^{(3)}$): $\eta \propto 1/nd$
Slim KANs
The authors also introduce Slim KANs, which apply this low-tensor-rank structure from day one (initializing the base tensor to zero). This creates a permanently compressed KAN architecture that inherently mitigates overfitting through low-rank regularization.
Datasets
The authors evaluated LoTRA and Slim KANs across two primary domains:
- Physics-Informed PDE Benchmarks: Synthetic datasets representing three fundamental classes of second-order PDEs with varying physical parameters ($\lambda$):
- Elliptic equations (steady-state systems)
- Allen-Cahn equations (nonlinear parabolic equations describing phase separation)
- Hyperbolic equations (wave propagation models)
- Standard Image Classification: MNIST and CIFAR-10 (the latter integrated with a 1-layer ResNet) to test the generalization of Slim KANs outside of pure scientific computing.
Key Insights and Ideas
- Intrinsic Low-Rank Structure: The authors empirically verify that both pre-trained KAN parameters and their fine-tuning updates naturally exhibit a low Tucker rank structure (as shown by the rapid decay of singular values in Figure 1). This justifies why tensor compression works so well without destroying network expressiveness.
- The Multi-LR Necessity: In matrix-based LoRA, a single learning rate usually suffices. In tensor-based LoTRA, the mathematical interplay between modes means that ignoring the dimensionality of the input and basis functions causes gradient explosion or stagnation.
- Implicit Regularization: When training Slim KANs on noisy data, the authors observed a distinct U-shaped validation curve relative to the compression ratio. Tightening the low-rank constraint effectively filters out high-frequency noise, acting as an excellent regularizer.
Key Results
- PDE Accuracy: LoTRA consistently matched or outperformed full parameter fine-tuning ("full") and training from scratch ("zero"), while drastically outperforming standard MLP baselines.
For instance, look at the relative error percentages for elliptic equations under different parameter scales:

- Massive Storage Savings: With a compression ratio ($cr$) of $1/4$, each mode of the tensor is quartered, resulting in a parameter footprint for the adapted layers that is $(1/4)^3 = 1/64$ (less than 2%) of the original size.
- Vision Tasks: While KANs historically lag slightly behind MLPs in pure computer vision tasks, Slim KANs combined with a ResNet on CIFAR-10 achieved up to 76.5% accuracy, outperforming comparable MLP-Small baselines (73.29%) while utilizing significantly fewer parameters.
Limitations and Future Work
- Scope of Tasks: The primary focus of this work is heavily weighted toward physics-informed machine learning (PDEs). The effectiveness of LoTRA on large-scale computer vision or natural language processing tasks remains an open question.
- Theoretical Simplifications: The optimal learning rate proofs were derived using a simplified one-layer toy case before being generalized to multi-layer networks via corollaries. A deeper, holistic theoretical analysis of multi-layer tensor interactions during gradient descent is a clear next step.
Summary updated