Trainable Rational Activation Functions

19 concepts16 papers1 proposal

In deep learning, the choice of activation function has traditionally been a manual, architectural decision. For years, practitioners have relied on fixed, piecewise linear functions like ReLU, or smooth approximations like GELU and Swish, applying them uniformly across entire networks. However, a fixed activation function forces every neuron to transform its input space in the exact same way, regardless of the layer's depth or the task's specific geometry.

A powerful alternative is to let the neural network learn its own activation functions during training. By parameterizing activation functions as rational functions—the ratio of two low-degree polynomials—networks can dynamically discover the optimal non-linearity for every layer. This approach drastically increases a network's expressive capacity with negligible parameter overhead, offering profound advantages in mathematical optimization, parameter-efficient fine-tuning, and physical simulations.

The Intuition: Why Ratios Excel

To understand why rational activations work, it is helpful to contrast them with standard polynomial expansions (like Taylor series). If we want to approximate a non-linear shape that contains both flat regions and sharp transitions (such as a ReLU curve), a standard polynomial performs poorly. To capture a sharp bend, a polynomial requires a very high degree, which inevitably introduces wild oscillations elsewhere—a behavior known as Runge's phenomenon.

Standard Polynomial (Taylor): High degree required -> Oscillates wildly
Rational Function (P(x)/Q(x)): Low degree required  -> Smooth transitions & flat regions

A rational function $R(x) = \frac{P(x)}{Q(x)}$ circumvents this limitation entirely. Because it features a polynomial in the denominator, it can model sharp transitions, asymptotes, and flat saturation zones using incredibly low degrees (typically degree 3 to 5). As the denominator $Q(x)$ varies, the function can rapidly change its slope or flatten out, allowing it to mimic virtually any standard activation function or evolve into entirely novel shapes.

As Molina et al. demonstrated with Padé Activation Units (PAUs), these learnable ratios provide a mathematically more efficient representation than classical expansions. They allow a network to adaptively find the optimal smooth activation shape for a specific task, combining the flexibility of non-linear modeling with the stability of low-degree polynomials.

The Expressivity Advantage

The theoretical foundation for this approach lies in approximation theory. Telgarsky proved that rational functions can approximate non-smooth activations like the absolute value function with an approximation error that decays exponentially with the degree of the function. This stands in stark contrast to standard polynomials, which require exponentially more resources to achieve the same accuracy.

When integrated into neural networks, this property translates directly into structural efficiency:

  • Depth Reduction: Boullé et al. proved that rational neural networks can approximate smooth functions with exponentially less depth than standard ReLU networks. A small set of shared polynomial coefficients can represent high-degree rational functions through layer composition, maximizing expressive power without inflating the model size.
  • Architecture Stabilization: Tang et al. formalized these expressivity advantages, showing that trainable rational fractions allow models to optimize their internal activation shapes on the fly. Remarkably, this added flexibility can even stabilize complex architectures like Transformers when traditional normalization layers are completely omitted.

Dynamic Adaptation and Fine-Tuning

Because rational activations are parameterized by the coefficients of their numerator and denominator polynomials, these coefficients can be updated via standard backpropagation alongside the network's weights. This makes them highly effective tools for adapting large, pretrained models to new tasks without the cost of full fine-tuning.

In the context of Parameter-Efficient Fine-Tuning (PEFT), Yin et al. introduced NoRA, a method that leaves weight matrices largely frozen and instead adapts the model's non-linearities using learnable rational functions. This approach demonstrates that modifying the activation behavior directly can match or exceed the performance of full fine-tuning while updating only a minuscule fraction of the parameters.

Similarly, Moosavi et al. developed Adaptable Adapters (AA), which replace fixed non-linearities with learnable polynomial ratios. This allows the network to compress its adapter layers by roughly 50% while maintaining high accuracy in data-scarce environments.

This paradigm of layer-specific activation tuning has been successfully applied across a variety of architectures:

  • Transformers: Fang et al. proposed the Rational Activation Function Transformer (RAFT), replacing fixed formulas like GELU with trainable rational functions to discover optimal activation curves per layer and per task.
  • Prompt Tuning: Zhu et al. utilized Padé Activation Units within instance-aware prompt tuning frameworks to dynamically optimize non-linearities for large language models.
  • Regression Heads: For complex psychometric tasks, such as predicting item difficulty and student response times, the authors in Gombert et al. and Baldwin et al. leveraged rational activation functions in specialized regression heads to capture highly non-linear human behavior patterns.

Inductive Biases for Physical Systems

One of the most compelling use cases for rational neural networks is in scientific computing and the simulation of dynamical systems. Standard piecewise linear activations like ReLU introduce non-differentiable "kinks" or sharp corners into the network's prediction surface. When solving differential equations, these artificial discontinuities break down because physical trajectories are inherently continuous and smooth.

Rational functions provide an ideal inductive bias for physics because they are smooth, infinitely differentiable, and capable of capturing complex mathematical singularities. Huang et al. demonstrated that incorporating rational activations into neural network correctors allows numerical solvers to take much larger time steps, accelerating simulations of dynamical systems by orders of magnitude without sacrificing physical accuracy. Similarly, Zhang et al. utilized trainable rational activations within stage-correction neural networks to significantly improve the stability of time integrators on complex non-linear systems.

Beyond pure simulation, the mathematical structure of a rational function can make a neural network interpretable. In a standard network, the weights are an opaque "black box." However, because rational functions possess explicit mathematical poles (points where the denominator $Q(x) = 0$), they can be mapped directly to physical phenomena.

Input Data ---> Rational Neural Network ---> Learned Activation Ratios
                                                       |
                                            Denominator Q(x) = 0
                                                       |
                                                       v
                                         Identifies Physical Shock/Singularity

Boullé leveraged this property to discover human-understandable Green's functions from data, showing that the poles of the learned rational activations naturally align with the exact physical locations of shocks and singularities in the underlying physical system.

Whether used to enforce continuous templates in symbolic regression (Scholl et al.), verify safety boundaries via polynomial arithmetic (Huang et al.), or compress deep language models, rational activation functions bridge the gap between fixed architectural constraints and fluid, data-driven optimization.

Key Foundational Work

Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks (2020) — contributed 2 concept elements to this direction.
Rational neural networks (2020) — contributed 2 concept elements to this direction.
Neural Networks and Rational Functions (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_rational_activation_functions_f890e519,
  title = {rational activation functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rational-activation-functions/rdr_f890e519c9a94643ae1f27c8a7095376},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_rational_activation_functions_f890e519_v1,
  title = {rational activation functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rational-activation-functions/rdr_f890e519c9a94643ae1f27c8a7095376/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake