Learnable Activation Functions for Plastic Neural Networks

174 concepts144 papers1 proposal

In standard deep learning, we treat the architecture of a neural network as a rigid split of responsibilities. The weights and biases ($W$ and $b$) are dynamic parameters learned from data, while the activation function ($\sigma$) is a static, hand-designed hyperparameter chosen before training begins. Whether it is ReLU, GELU, or a sinusoidal wave, this function applies the exact same non-linear transformation to every neuron, across every layer, regardless of the input distribution.

This "one-size-fits-all" approach introduces severe architectural rigidity. If a network needs to capture a highly complex or localized non-linear relationship, it cannot adapt the activation function itself. Instead, it must rely on increasing model width or depth so that combinations of linear transformations can approximate the desired shape. As noted by Agostinelli et al. in Learning Activation Functions to Improve Deep Neural Networks, this constraint forces a heavy reliance on over-parameterization, driving up hardware requirements and computational costs.

To bypass this limitation, a growing body of research focuses on learnable and adaptive activation functions. By giving individual neurons or layers the plasticity to shape their own non-linearities, models can achieve higher expressivity, better parameter efficiency, and superior generalization.


Parameterizing the Non-Linearity

The core challenge of a learnable activation function is parameterization: how do we allow the function's shape to change during training without introducing optimization instability or an explosion of new parameters?

Basis Expansions and Custom Shapes

One intuitive approach is to construct the activation function from a combination of flexible base functions. For example, Eisenach et al. in Nonparametrically Learning Activation Functions in Deep Neural Nets leveraged a Fourier basis expansion. By allowing each node to nonparametrically "shape" its own activation curve, the network can adapt its internal transformations directly to the underlying structure of the target function.

Unified Parametric Families

Another strategy unifies existing, popular activation functions into a single mathematical formula controlled by learnable parameters. In Activate or Not: Learning Customized Activation, Ma et al. introduced the ACON (Activate Or Not) family. ACON unifies Maxout-style functions like ReLU and Swish, allowing the network to explicitly learn whether individual neurons should behave linearly or non-linearly.

Similarly, Deng et al. proposed Adaptive Parametric Activation (APA), which unifies Sigmoid, Gumbel, ReLU, and GELU. By optimizing these parameters alongside the weights, the network automatically tunes its activation shapes to match the statistical priors of the input data, which is particularly effective for highly imbalanced or long-tailed datasets.

Rational Functions

Rational neural networks use trainable fractions of polynomials as their activation functions. The mathematical advantage here is profound: Tang et al. demonstrate in Rational Neural Networks have Expressivity Advantages that rational activations provide an exponential expressivity advantage over fixed activations like ReLU.

Because a rational function can approximate smooth curves with far fewer parameters, it drastically improves parameter efficiency. Fang applied this to language models in Transformers with Learnable Activation Functions, showing that allowing a Transformer to adapt its activation shape per layer significantly improves performance, especially in data-scarce environments where standard models overfit.


Context-Aware and Input-Dependent Non-Linearities

While learning a static activation shape during training is powerful, an even more flexible paradigm involves dynamic adaptation, where the activation function changes its behavior during inference based on the input data itself.

Biological neurons do not have fixed firing thresholds; they vary their sensitivity based on their environment and the signals they receive. To replicate this behavior, Lee et al. proposed Adaptive SwisH (ASH) in Stochastic Adaptive Activation Function. ASH dynamically samples informative features based on the distribution of neuron outputs, allowing the activation threshold to shift stochastically depending on the input context.

Static Activation (e.g., ReLU)
Input X ───> [ Fixed Curve ] ───> Output Y

Dynamic Activation (e.g., FleS)
Input X ───> [ Curve Shape Adjusted by Context/Batch Stats ] ───> Output Y

On an architectural scale, standard self-gated activations (like Swish or GELU) often suffer from "non-local tension"—a phenomenon where activations saturate and fail to utilize rich contextual information across different channels. To resolve this, Sudong Cai et al. developed FleS (Flexible Scaling) in Toward Principled Flexible Scaling for Self-Gated Neural Activation. FleS adaptively scales the steepness and range of activation curves based on channel-wide statistics, ensuring that neurons maintain high discriminative power even when processing highly variable input distributions.


Geometric Control and Model Steering

Modulating the activation function fundamentally changes the geometry of a neural network's decision boundary. In traditional fine-tuning, adapting a model to a new task requires updating millions of weight parameters to shift features in high-dimensional space. Learnable activations offer an alternative: steering the model by bending the decision boundary instead.

In Curvature Tuning: Provable Training-free Model Steering From a Single Parameter, Hu et al. demonstrate that injecting a single hyperparameter into the activation functions of a pretrained model can steer its behavior.

Instead of shifting features, this method modulates the curvature of the decision boundary. This training-free approach achieves massive gains in adversarial robustness and generalization with a fraction of the parameters used by traditional parameter-efficient fine-tuning (PEFT) methods like LoRA.


Flipping the Paradigm: Kolmogorov-Arnold Networks

The ultimate extension of this research direction is to completely invert the roles of weights and activations. Traditional Multi-Layer Perceptrons (MLPs) place fixed activation functions on nodes and learnable weights on edges. Kolmogorov-Arnold Networks (KANs) flip this paradigm by placing learnable activation functions directly on the edges and eliminating fixed node activations entirely.

Multi-Layer Perceptron (MLP)
Node (Fixed Activation) ─── Edge (Learnable Weight) ───> Node (Fixed Activation)

Kolmogorov-Arnold Network (KAN)
Node (Summation) ─────── Edge (Learnable Activation) ───> Node (Summation)

The authors of U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (Li et al.) demonstrated that replacing standard linear layers in deep bottlenecks with KANs allows the network to learn highly complex, heterogeneous representations with superior parameter efficiency.

To make this practical for downstream adaptation, Xuan Cai et al. introduced LoKi in LoKi: Low-dimensional KAN for Efficient Fine-tuning Image Models. LoKi utilizes a low-dimensional bottleneck architecture to replace the fixed activations of standard adapters with learnable KANs. This architectural plasticity allows the model to achieve excellent generalization on small datasets or under severe distribution shifts, proving that letting the activation function learn is often far more efficient than simply making the network bigger.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 144 papers
  2. Version 1 · 2026-10-04 · 240 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_static_activation_functions_1891b099,
  title = {static activation functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-activation-functions/rdr_1891b09914fd44f488dff2810cd8d433},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_static_activation_functions_1891b099_v2,
  title = {static activation functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-activation-functions/rdr_1891b09914fd44f488dff2810cd8d433/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake