In standard deep learning, activation functions like ReLU or Tanh are treated as fixed architectural constants. While these "set it and forget it" components work well for general classification, they often struggle with the precision required for regression, the complexity of physical singularities, or the efficiency needed for compact models.
A growing body of research suggests that we should instead treat the activation function as a learnable parameter. By replacing fixed nonlinearities with Rational Activation Functions—the ratio of two trainable polynomials—networks can adapt their internal transformations to the specific geometry of the data.
The Power of Rational Approximation
The core intuition behind this shift lies in approximation theory. Most traditional activations are either piecewise linear (ReLU) or smooth but rigid (Sigmoid). When a network needs to approximate a complex function, it must stack many layers of these simple units to mimic the target shape.
Rational functions, defined as $f(x) = \frac{P(x)}{Q(x)}$ where $P$ and $Q$ are polynomials, are mathematically more "flexible" than polynomials of the same degree. As Alejandro Molina and Kristian Kersting demonstrate with Padé Activation Units (PAUs), these functions can approximate any standard activation or discover entirely new ones during training Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks.
The theoretical advantage is significant: Matus Telgarsky proved that rational functions can approximate non-smooth features, like the absolute value function, with an error that decays exponentially relative to the function's degree Neural Networks and Rational Functions. This allows a single rational layer to achieve the same expressive power as a much deeper ReLU network.
Capturing Physical Singularities
One of the most compelling applications of rational activations is in scientific computing and physics-informed neural networks (PINNs). Physical systems often involve "shocks," boundary layers, or singularities where values approach infinity or change abruptly.
Standard neural networks struggle with these discontinuities because their activations are globally smooth or bounded. Rational functions, however, have poles—points where the denominator $Q(x)$ approaches zero. Nicolas Boullé and Alex Townsend have shown that rational networks can "discover" physical laws by positioning these poles near the actual singularities of a system, such as shocks in fluid dynamics Data-driven discovery of Green’s functions with human-understandable deep learning.
This makes the network's internal parameters physically interpretable. Instead of a "black box," the model becomes a diagnostic tool: the location of the learned poles tells the researcher exactly where the critical behaviors in the physical system occur Neural network-based singularity detection and applications.
Scaling Kolmogorov-Arnold Networks
The recent interest in Kolmogorov-Arnold Networks (KANs)—which place learnable functions on the edges (weights) rather than the nodes—has further highlighted the utility of rational functions. Original KANs relied on B-splines, which are computationally expensive and difficult to scale on GPUs.
Researchers like Lizhong Chen have addressed this by replacing splines with rational functions in the Kolmogorov-Arnold Transformer (KAT) Kolmogorov-Arnold Transformer. This substitution, often called Group-Rational KAN (GR-KAN), provides the same high expressivity as splines but with much better numerical stability and training efficiency FlashKAT: Understanding and Addressing Performance Bottlenecks in the Kolmogorov-Arnold Transformer. By using Horner’s method to evaluate the polynomials, these models maintain high-order accuracy while significantly reducing the number of required multiplications.
Efficiency through Adaptability
Beyond specialized physics tasks, rational activations offer a path toward more efficient general-purpose models. In regression tasks, Le Hou and Dimitris Samaras introduced Smooth Adaptive Activation Functions (SAAFs), which use piecewise polynomials to fit complex non-linear mappings with minimal parameter overhead Neural Networks with Smooth Adaptive Activation Functions for Regression.
Similarly, in the realm of Large Language Model (LLM) fine-tuning, the NoRA (Nonlinear Rational Adapter) approach suggests that instead of only updating weight matrices (as in LoRA), we should also adapt the activation functions Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning. By making the nonlinearity itself learnable through rational functions, models can adapt to new domains with as little as 0.4% of the original parameters.
Whether the goal is to solve partial differential equations with surgical precision or to compress a vision transformer for edge devices, the move from fixed to rational activations represents a fundamental shift: letting the network decide not just how much to fire, but how to fire.