Why do neural networks use specific activation functions like ReLU, Sigmoid, or Tanh? In traditional deep learning, these functions are often chosen heuristically based on empirical performance or optimization convenience. However, a growing body of research reveals a deeper, more principled foundation: activation functions can be derived directly from the statistical properties of the data distributions the network is trying to model.
By aligning the mathematical form of a network's nonlinearities with the underlying data distribution—such as exponential families or mixture models—neural networks cease to be arbitrary black boxes. Instead, their forward passes and training dynamics directly emulate classical statistical inference algorithms.
Activations as Inverse Link Functions
In classical statistics, generalized linear models use a link function to map a linear predictor to the expected value of a target data distribution. This link function is fundamentally tied to the distribution's log partition function. Recent work demonstrates that neural network activation functions can be systematically derived using this exact relationship, rather than selected by trial and error.
In Deep equilibrium models as estimators for continuous latent variables, Tsuchida et al. demonstrate that common activation functions, such as the sigmoid, are the direct mathematical consequence of specific statistical assumptions regarding the probability distribution of the data. By re-parameterizing exponential family link functions, as explored by the authors in Stochastic gradient updates yield deep equilibrium kernels, these statistical operators can serve directly as activation functions within a neural network framework. This ensures that the network's structural nonlinearities are mathematically aligned with the expected noise model of the input data.
Equilibrium Networks and MAP Inference
This statistical alignment becomes particularly powerful in implicit deep learning, such as Deep Equilibrium Models (DEQs). Unlike traditional networks that pass inputs through a fixed sequence of layers, a DEQ computes its output by iterating a single layer until it reaches a fixed-point equilibrium.
When the activation function of a DEQ layer is designed as the inverse link function of an exponential family distribution, the network's forward pass gains a clear statistical purpose. As shown by Tsuchida et al. in When are equilibrium networks scoring algorithms?, Maximum A Posteriori (MAP) estimation for exponential family models can be directly reformulated as solving a fixed-point equation.
Consequently, the fixed-point iteration of the DEQ layer becomes an optimization process that solves for the MAP estimates of latent variables. By matching the first-order optimality conditions of a statistical objective with the network's equilibrium equation, the model's output functions as a principled statistical estimator rather than a heuristic approximation.
Geometric Signatures and Density Factorization
Deriving activations from distributions also explains why deep networks are remarkably efficient at representing complex probability densities. In f-GANs in an Information Geometric Nutshell, Nock et al. analyze "strongly admissible" activation functions—defined as convex, strictly increasing, and continuously differentiable functions.
The authors prove that standard activations like ReLU and Softplus act as geometric signatures for specific deformed exponential families. When a network utilizes these activations, it naturally factorizes the complex probability density of the data. This structural property provides an information-geometric explanation for the power of deep architectures: it allows them to break down and represent intricate probability distributions far more efficiently than shallow networks.
Matching Score Functions to Automate Classical Algorithms
The principle of distribution-derived activations extends to modern generative modeling, such as Denoising Diffusion Probabilistic Models (DDPMs). When a diffusion model learns a complex distribution, it trains a neural network to predict the score function—the gradient of the log-density of the data.
If the architecture of the neural network is explicitly tailored to match the functional form of the target distribution's score function, the training process simplifies dramatically. In Learning Mixtures of Gaussians Using the DDPM Objective and its corresponding conference publication Learning Mixtures of Gaussians Using the DDPM Objective, Shah et al. study the task of learning Gaussian Mixture Models (GMMs).
The authors demonstrate that a two-layer neural network utilizing hyperbolic tangent (tanh) activations perfectly mirrors the mathematical structure of a Gaussian mixture's score function. Because the network's nonlinearity is explicitly tuned to the target distribution, gradient descent on the standard DDPM objective is mathematically guaranteed to be highly efficient. Rather than performing a blind search, the score-matching optimization process implicitly automates and emulates classical statistical algorithms, combining spectral methods and the Expectation-Maximization (EM) algorithm.
Conclusion
Grounding activation functions in statistical distributions provides a powerful framework for neural network design. Whether interpreting a DEQ fixed-point as a MAP estimator, a GAN layer as an information-geometric density factorizer, or a diffusion model as an automated EM solver, deriving nonlinearities from data distributions bridges the gap between heuristic deep learning and rigorous statistical inference.
Key Foundational Work
When are equilibrium networks scoring algorithms? (2022) — contributed 6 concept elements to this direction.
f-GANs in an Information Geometric Nutshell (2017) — contributed 2 concept elements to this direction.