Shifting Activation Functions from Nodes to Edges

237 concepts166 papers4 proposals

For decades, the Multi-Layer Perceptron (MLP) has served as the foundational building block of deep learning. The structural formula of an MLP is deeply familiar: a node receives a weighted sum of inputs from the previous layer, passes that sum through a fixed non-linear activation function (such as ReLU or Sigmoid), and transmits the result forward. To learn increasingly complex functions, an MLP relies on architectural depth—stacking layers to progressively slice high-dimensional space with flat hyperplanes.

A rapidly growing body of research explores a fundamental alternative to this paradigm: the Kolmogorov-Arnold Network (KAN). As formalized by Ziming Liu et al. in KAN: Kolmogorov–Arnold Networks, KANs completely invert the traditional allocation of labor. Instead of fixed activation functions at the nodes and scalar weights on the edges, KANs place learnable univariate activation functions directly on the edges. The nodes themselves perform no non-linear transformations; they simply sum the incoming signals.

Traditional MLP Node:
Inputs (x₁, x₂) ──> [ Weights (w₁, w₂) ] ──> [ Sum (∑) ] ──> [ Fixed Activation (σ) ] ──> Output

Kolmogorov-Arnold Network (KAN) Node:
Inputs (x₁, x₂) ──> [ Learnable Functions (f₁(x₁), f₂(x₂)) ] ──> [ Sum (∑) ] ──> Output

By shifting the complexity of function approximation from static node hierarchies to flexible, edge-based transformations, this architecture introduces a completely different inductive bias—one that yields remarkable parameter efficiency, mathematical interpretability, and precision in scientific modeling.

The Intuition: Why Edge-Based Functions Work

The theoretical foundation of this architecture traces back to the Kolmogorov-Arnold representation theorem. The theorem states that any multivariate continuous function can be decomposed into a finite composition of continuous univariate functions and addition.

To understand why this is powerful, consider how a standard MLP approximates a smooth curve. A ReLU-based network must piece together thousands of tiny linear segments, requiring significant depth and parameter counts to minimize error. If you want to increase the precision of an MLP, you must add more layers or wider hidden dimensions, which scales the parameter count aggressively.

In contrast, a KAN parameterizes each edge function using flexible, localized curves—most commonly B-splines, power units, or polynomials. This allows a single connection between two nodes to adaptively fit intricate non-linear relationships. Instead of expanding the entire network architecture to achieve higher precision, a KAN can undergo "grid refinement," making the underlying spline grids finer without altering the network's macro-structure.

The theoretical viability of this approach has been rigorously validated. For instance, Xianyang Zhang et al. demonstrated in Generalization Bounds and Model Complexity for Kolmogorov–Arnold Networks that a KAN's generalization complexity scales based on the smoothness and coefficients of its learnable edge functions rather than the sheer quantity of hidden nodes. Furthermore, Yixuan Wang et al. proved in On the expressiveness and spectral bias of KANs that while an MLP can theoretically map to an edge-based architecture, representing fine-grained spline functions within a standard MLP requires a parameter count that scales poorly with the resolution of the spline grid.

This concept builds upon a long history of adaptive activation functions. Early precursors explored giving individual neurons unique, learnable parameters. Matthew Hoffman in Learning Activation Functions to Improve Deep Neural Networks proposed optimizing sums of hinge functions per neuron via gradient descent, while Qinliang Su et al. in A Probabilistic Framework for Nonlinearities in Stochastic Neural Networks showed that allowing individual units to learn unique activation shapes significantly improves pattern capture compared to uniform layer-wide activations. KANs take this a step further by decentralizing the activation function entirely, moving it from the neuron to every individual connection.

Overcoming the Computational Bottleneck

Despite their expressive power, original KAN formulations suffer from a major practical drawback: they are slow. Modern hardware accelerators (GPUs) are highly optimized for dense, uniform matrix multiplications. Evaluating recursive B-splines independently across thousands of network edges breaks this parallelism, creating severe computational bottlenecks.

To bridge the gap between theoretical expressivity and hardware efficiency, researchers have introduced several optimized variants:

Opening the Black Box: Built-In Interpretability

One of the most compelling reasons to shift activations to the edges is transparency. In a standard MLP, a feature's influence is obscured as it passes through a dense web of weights and shared activations. In a KAN, because each connection possesses its own isolated univariate function, the internal logic of the network can be directly inspected.

Because these edge functions are one-dimensional curves, they can be plotted and visualized. Over the course of training, many edge functions naturally flatten out or simplify, allowing researchers to prune unnecessary connections and even distill the remaining curves into explicit symbolic mathematical expressions.

Input Feature (x) ───[ Learned Curve: f(x) = sin(x) ]───> Output Node

This explicit representation has driven adoption across domains where understanding why a model makes a decision is as critical as the decision itself:

Expanding the Architecture Ecosystem

The paradigm of learnable edge activations is rapidly expanding beyond simple tabular function fitting into structured data domains, replacing MLPs within broader deep learning frameworks.

Graph Learning

Traditional Graph Neural Networks (GNNs) rely on MLPs to transform messages passed between neighboring nodes. By swapping MLPs for KANs, the authors of KAGNNs: Kolmogorov-Arnold Networks meet Graph Learning (Roman Bresson et al.) demonstrated a massive leap in graph regression tasks, as flexible edge functions excel at fitting the smooth, continuous multivariate regularities common in scientific graph data. To enforce geometric constraints, Ran Elbaz et al. introduced FS-KAN: Permutation Equivariant Kolmogorov-Arnold Networks via Function Sharing, sharing univariate functions across edges to achieve strict permutation equivariance. Additionally, Taoran Fang et al. developed KAA: Kolmogorov-Arnold Attention for Enhancing Attentive Graph Neural Networks, replacing linear attention scoring with KANs to achieve near-infinite expressive power in ranking neighbor importance.

Continuous Fields and Physics

Modeling physical systems requires capturing sharp gradients and continuous spaces. Jin Lee et al. developed KANO: Kolmogorov-Arnold Neural Operator to overcome the limitations of Fourier Neural Operators when dealing with position-dependent physics. For complex partial differential equations, Changhong Mou et al. utilized edge-based splines in Energy-Dissipative Evolutionary Kolmogorov-Arnold Networks for Complex PDE Systems to provide the localized flexibility and smoothness needed to represent sharp spatial gradients effectively.

Vision and Parameter-Efficient Fine-Tuning

KANs are also proving to be highly effective as lightweight adapters for massive vision models. Xuan Cai designed LoKi: Low-dimensional KAN for Efficient Fine-tuning Image Models, a bottleneck architecture that replaces fixed activations in standard adapters with learnable B-splines, achieving superior task generalization with minimal memory overhead. Similarly, Ruichao Hou et al. built KAN-SAM: Kolmogorov-Arnold Network Guided Segment Anything Model for RGB-T Salient Object Detection, updating less than 1% of total parameters by injecting thermal features through KAN adapters. For specialized segmentation tasks, Daniele Rege Cambrin et al. integrated KANs into a U-Net structure (KAN You See It? KANs and Sentinel for Effective and Explainable Crop Field Segmentation), improving boundary detection accuracy while cutting computational costs nearly in half.

By rethinking the fundamental structure of the neuron, this research direction demonstrates that the way a network learns non-linearities is just as critical as its architectural depth. Shifting activation functions to the edges provides a mathematically grounded path toward models that are not only smaller and more precise, but fundamentally open to human inspection.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 161 papers
  2. Version 1 · 2026-10-04 · 144 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_kolmogorov_arnold_networks_966427f1,
  title = {Kolmogorov-Arnold Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kolmogorov-arnold-networks/rdr_966427f11b3344a1aadb8ed9a3a8552e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_kolmogorov_arnold_networks_966427f1_v2,
  title = {Kolmogorov-Arnold Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kolmogorov-arnold-networks/rdr_966427f11b3344a1aadb8ed9a3a8552e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake