For decades, the Multi-Layer Perceptron (MLP) has been the foundational building block of deep learning. In a standard MLP, we pass data through fixed activation functions (like ReLU or GeLU) located at the "nodes" and learn the linear weights on the "edges" connecting them. Ziming Liu and colleagues recently introduced a fundamental shift to this architecture with KAN: Kolmogorov–Arnold Networks, which moves the learning process from the weights to the activation functions themselves.
Flipping the Architecture
The intuition behind a Kolmogorov-Arnold Network (KAN) is a reversal of the MLP paradigm. Instead of fixed activations at nodes and learnable weights on edges, KANs use learnable univariate functions on the edges and simple summation at the nodes.
Mathematically, this is inspired by the Kolmogorov-Arnold representation theorem, which states that any multivariate continuous function can be represented as a finite composition of continuous functions of a single variable and the operation of addition. By parameterizing these univariate functions—typically using B-splines—the network can adapt the shape of its non-linearities to fit the data precisely.
As Jin Lee and Ziming Liu demonstrate in KANO: Kolmogorov-Arnold Neural Operator, this shift allows the network to overcome the "spectral bottleneck" of traditional Fourier methods. Because the functions on the edges are flexible, the model can jointly parameterize spatial and spectral bases, leading to extreme parameter efficiency.
Why It Works: Expressivity and Efficiency
The primary advantage of KANs is their ability to model complex, high-dimensional non-linearities with significantly fewer parameters than MLPs.
- Parameter Efficiency: Because each edge is an expressive spline rather than a single scalar weight, a very small KAN can often outperform a much larger MLP. For instance, Ruoxi Chen showed in Transformer-KAN that replacing feedforward layers with KANs achieved higher accuracy in seismic classification while using 60% fewer parameters.
- Data Efficiency: KANs excel in low-data regimes. Hao Hao and Aimin Zhou found that KANs serve as superior surrogate models for evolutionary algorithms because they capture the underlying function landscape more accurately from sparse data points than traditional networks.
- Grid Refinement: Unlike MLPs, KANs can be made more accurate without retraining from scratch. By simply increasing the resolution of the underlying spline grids (fine-grained refinement), the model can learn more intricate details of a function.
From Black Box to Symbolic Math
One of the most compelling facets of this research direction is interpretability. In a standard neural network, the "knowledge" is buried in millions of abstract weights. In a KAN, the knowledge is stored in one-dimensional curves that can be visualized and even simplified into symbolic mathematical expressions.
In scKAN, Haohuai He and Fan Xu utilized these "learnable activation curves" to identify the biological drivers of disease. Because the relationship between an input feature and the output is explicitly modeled by a spline on an edge, researchers can plot that spline to see exactly how a specific gene influences a cell-type classification.
This transparency has led to a surge of "white-box" AI applications in the sciences. Joshua Fan and Carla P Gomes used this property in ScIReN to discover hidden physical laws in the Earth's carbon cycle, effectively turning a neural network into a tool for symbolic discovery.
Integration Across Domains
The "KAN-based" approach is rapidly being integrated into existing architectures to solve domain-specific problems:
- Computer Vision: Chenxin Li introduced U-KAN, which replaces the linear bottlenecks of U-Nets with KAN layers, improving medical image segmentation and generation.
- Graph Learning: Fan Zhang developed GraphKAN, using splines to enhance feature extraction in graph neural networks, which helps mitigate the "over-smoothing" problem where node features become indistinguishable.
- Reinforcement Learning: Zhen Tian applied KANs to autonomous driving in K-DQN, finding that the spline-based architecture better represents the intricate value landscapes required for complex maneuvers like navigating roundabouts.
Theoretical Constraints
Despite their success, KANs are not a universal panacea. Fan Yang and Kaixiong Zhou have investigated Catastrophic Forgetting in Kolmogorov-Arnold Networks, noting that while the local nature of splines helps with memory in low dimensions, KANs still suffer from a "curse of dimensionality" in high-dimensional tasks where activation supports begin to overlap. Additionally, the computational cost of evaluating splines can be higher than simple matrix multiplications, a trade-off for their superior parameter efficiency.