Incorporating Bias Parameters into Deep Learning Theory

This direction is merged into another direction: architectural homogeneity constraint (model homogeneity assumption). Its last version stays readable and citable here.

Version 1 of 129 papers29 conceptsfrozen 2026-10-04

In the mathematical study of neural networks, there is a common, unspoken agreement: we often pretend the "bias" term ($b$ in $Wx + b$) doesn't exist. By assuming networks are purely linear transformations followed by activations, researchers can leverage elegant symmetries, simplify optimization landscapes, and make high-dimensional integrals tractable.

However, a growing body of research suggests that this simplification—while mathematically convenient—creates a significant "theory-practice gap." As Andre Manoel and colleagues noted in Entropy and mutual information in models of deep neural networks, excluding learnable biases means our models fail to capture the behavior of standard neurons used in the real world. This research direction focuses on re-integrating the bias term into our theoretical understanding of generalization, optimization, and representation.

The Mathematical Convenience of Zero Bias

Why do theorists omit the bias? In many frameworks, such as the Neural Tangent Kernel (NTK) or spectral analysis, the absence of bias allows the network to be treated as a homogeneous function. For instance, Quynh Nguyen and Guido Montúfar point out in Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks that excluding bias is often necessary to maintain tractability when calculating kernel eigenvalues.

Similarly, in the study of convex geometry, Tolga Ergen and Mert Pilanci explain that while their convex analytic framework provides deep insights into why networks don't overfit, the primary proofs often exclude bias to simplify the underlying derivations in Convex Geometry of Two-Layer ReLU Networks. Without bias, every ReLU "kink" must pass through the origin; with bias, those kinks can move anywhere in the input space, fundamentally changing the geometry of the solution.

The Functional Necessity of the Offset

The bias term is not just a minor offset; it is the mechanism that enables affine transformations. As Rong Ge and Rohith Kuditipudi argue in Learning Two-layer Neural Networks with Symmetric Inputs, omitting bias limits a model's ability to represent the basic shifts found in practical data.

In optimization, Arnulf Jentzen and Timo Kröger highlight that bias parameters are critical for shifting activation functions to the regions where they are most effective. Their work in Convergence rates for gradient descent... with biases emphasizes that excluding these parameters creates a discrepancy between mathematical proofs of convergence and the actual architectures that practitioners train.

Bias as a Structural Driver

Recent work has shown that bias terms do more than just shift functions; they actively shape the network's internal properties:

Closing the Theory-Practice Gap

The field is now moving toward "bias-aware" theories. Yixi Xu provided a breakthrough in Understanding Weight Normalized Deep Neural Networks with Rectified Linear Units by proving generalization bounds that hold even when bias neurons are present in every hidden layer. Similarly, Pranjal Awasthi and Aravindan Vijayaraghavan developed the first polynomial-time algorithms for learning ReLU networks that explicitly include bias terms in Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations.

By treating the bias term as a first-class citizen rather than a nuisance variable, these researchers are building a theoretical foundation that finally looks like the networks we actually build.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_incorporating_bias_parameters_into_deep__300f4687,
  title = {Incorporating Bias Parameters into Deep Learning Theory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incorporating-bias-parameters-into-deep-learning-theory/rdr_300f4687b306457ca153f5161c360f47},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_incorporating_bias_parameters_into_deep__300f4687_v1,
  title = {Incorporating Bias Parameters into Deep Learning Theory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incorporating-bias-parameters-into-deep-learning-theory/rdr_300f4687b306457ca153f5161c360f47/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}