Standard deep learning architectures treat activation functions as rigid, architectural hyperparameters. We carefully select ReLU, GELU, or Swish before training begins, and then apply that exact same geometric curve uniformly across every neuron, channel, and layer. This approach forces every part of a network to process information through an identical mathematical lens, regardless of its depth or the specific feature distribution it encounters.
The field of learnable activation functions rethinks this constraint. Instead of fixing these non-linear curves in advance, why not let the network optimize them alongside its weights? By treating the shape of the activation function as a trainable component, networks can customize how they bend and transform representation space, unlocking significant expressivity with virtually zero parameter overhead.
Why It Works: The Geometry of Non-Linearity
To understand why learnable activations are so effective, we must look at how neural networks manipulate data. A standard feedforward layer performs two primary operations:
- A linear transformation (matrix multiplication) that rotates, scales, and shifts the coordinate space.
- A non-linear activation function that bends, folds, or truncates the space.
When we use a fixed activation function like ReLU, the network's only way to adapt to highly complex, multi-modal, or non-smooth data distributions is to stack more linear layers. It must rely entirely on the matrix multiplications to position the data perfectly relative to the rigid bends of the fixed activation.
Learnable activation functions invert this dynamic. By allowing the activation curve itself to shift, sharpen, or even become multi-modal, a single neuron can capture complex decision boundaries that would otherwise require an entire sub-network of fixed units. This expands the network's hypothesis space far more efficiently than simply scaling width or depth.
Researchers have developed several distinct strategies to parameterize these flexible curves, ranging from simple piecewise lines to dynamic, input-dependent sub-networks.
Piecewise and Parametric Adjustments
The most straightforward way to make an activation function learnable is to parameterize its slopes, thresholds, or boundaries. Rather than a single fixed inflection point, the function is broken down into adjustable segments.
For instance, Zhou et al. introduce the Piecewise Linear Unit (PWLU), which constructs activations out of connected linear segments defined by adjustable boundary points and slopes. To balance this expressivity with training stability, Tavakoli proposes SPLASH (Simple Piecewise Linear and Adaptive with Symmetric Hinges). SPLASH constrains the activation function to remain continuous and grounded at the origin, but allows the slopes of its individual segments to evolve dynamically via gradient descent.
This structural flexibility is particularly valuable in highly constrained architectures, such as Binary Neural Networks (BNNs). Because BNNs compress weights and activations into 1-bit values, they often suffer from severe quantization noise and rigid feature representations. To counteract this, Liu et al. introduce learnable shift parameters before and after non-linear activations. This simple addition allows the network to explicitly reshape and shift its feature distributions, recovering much of the accuracy lost during binarization.
Smooth Functional Expansions and Rational Functions
While piecewise linear functions are computationally efficient, they introduce sharp kinks that can complicate gradient flow. A more sophisticated approach models activations using smooth, continuous mathematical families or basis expansions.
One highly effective formulation is the use of rational functions—the ratio of two polynomials. Molina et al. introduce Padé Activation Units (PAUs), where the coefficients of the numerator and denominator polynomials are integrated directly into the global backpropagation process. Because rational functions can approximate any continuous function to a high degree of accuracy, PAUs allow the network to either recreate standard activations or discover entirely novel, smooth shapes optimized for the specific dataset.
Alternatively, activations can be modeled nonparametrically using basis expansions, effectively treating the activation function as a small signal-processing problem:
- Fourier Expansions: Eisenach et al. model activation functions as a sum of sine and cosine waves. This allows individual neurons to optimize periodic or multi-modal transformations that would be impossible for standard functions to replicate.
- Kernel Expansions: Scardapane et al. introduce Kernel Activation Functions (KAFs), representing the neuron's response as a one-dimensional kernel expansion. By learning the mixing coefficients of fixed dictionary points, the network shapes its non-linearities directly from the data distribution.
Diffeomorphic Warping and Nested Sub-networks
For tasks demanding extreme precision—such as implicit neural representations (INRs), fluid dynamics modeling, or complex physics simulations—standard functional expansions may still lack the necessary representational capacity.
To overcome this, Chelly et al. propose DiTAC (Diffeomorphic Transformations for Activation Customization). Instead of replacing fixed activations, DiTAC applies smooth, invertible mathematical transformations (diffeomorphisms) to warp the input space before it passes through a base function like GELU. This vertical and horizontal warping allows existing architectures to retain the proven inductive biases of standard activations while gaining the ability to morph into highly complex, multi-modal geometries.
Taking this concept to its logical extreme, researchers have explored replacing the activation function entirely with a tiny, nested neural network. As demonstrated by Jo et al. in the context of implicit neural representations, these nested subnetworks allow individual neurons to evolve custom, highly intricate nonlinearities. This architectural flexibility enables models to capture fine-grained spatial details and high-frequency signals that rigid, fixed functions like sines or wavelets routinely smooth over.
Context-Dependent and Instance-Aware Activations
The ultimate evolution of learnable non-linearities is the transition from static learnable shapes (which freeze after training) to dynamic activations that adapt in real-time during inference based on the input data.
A prominent example of this is the ACON (Activate Or Not) family introduced by Ma et al.. ACON provides a unified framework that can smoothly transition between linear and non-linear behavior. By employing a tiny auxiliary meta-network, ACON learns to dynamically calculate a switching factor for each channel or pixel based on the input context, allowing the network to explicitly decide whether a feature requires non-linear processing.
Similarly, Lee et al. introduce instance-aware activations for binary networks via INSTA-BNN. Rather than using a fixed threshold for quantization, the activation function dynamically shifts and scales its parameters based on the real-time statistical profile (such as skewness) of the specific input sample.
By shifting the responsibility of non-linear adaptation from rigid architectural scaling to flexible, data-driven parameters, learnable activation functions provide a highly efficient path toward more compact, expressive, and robust neural networks.