Scale Equivariance through Bias-Free Architectures

This direction is split into other directions: bias-free homogeneous architecture. Its last version stays readable and citable here.

Version 1 of 156 papers95 conceptsfrozen 2026-10-04

Standard deep neural networks are notoriously "brittle" when they encounter data at scales different from their training set. A denoiser trained on mild noise often produces catastrophic artifacts when faced with heavy noise; a graph network trained on small clusters may fail on large ones.

Research across image restoration, high-dynamic-range (HDR) imaging, and algorithmic reasoning suggests that this failure stems from a simple architectural component: the additive bias term. By removing these biases, researchers have uncovered a class of bias-free networks that are inherently scale-equivariant, allowing them to generalize to intensities and noise levels far beyond their training distribution.

The Problem with Additive Biases

In a standard neural network layer, the operation is defined as $y = \sigma(Wx + b)$, where $W$ is a weight matrix, $b$ is a bias vector, and $\sigma$ is an activation function like ReLU. Mathematically, this makes the network a piecewise affine function.

The presence of $b$ anchors the network's logic to a specific absolute magnitude. As Sreyas Mohan demonstrates in Robust and interpretable blind image denoising via bias-free convolutional neural networks, these internal additive constants can "explode" or grow uncontrollably when the input magnitude $x$ deviates from the training range. This is why a standard denoiser might interpret high-intensity noise as a structural feature of the image rather than a signal to be removed.

Achieving Homogeneity

When all additive constants—including convolutional biases and batch normalization offsets—are removed, the network transforms from a piecewise affine function into a piecewise linear one. This architectural constraint enforces a mathematical property known as positive homogeneity:

$$f(\alpha x) = \alpha f(x) \text{ for } \alpha > 0$$

This means that if you scale the input (e.g., double the brightness or the noise magnitude), the output scales by exactly the same factor. This property, explored by Sébastien Herbreteau and Charles Kervrann in Normalization-Equivariant Neural Networks with Application to Image Denoising, ensures that the model's internal logic remains consistent regardless of the signal's global intensity.

Why Scale Equivariance Works

The intuition behind bias-free designs is that they force the network to learn relative relationships rather than absolute values.

1. Robustness to Unseen Noise

In image denoising, Sreyas Mohan shows that bias-free CNNs can generalize to noise levels $10\times$ higher than those seen during training. Because the network is homogeneous, it treats noise magnitude as a linear factor. This prevents the "performance collapse" typically seen in standard models. This principle is further extended in UDVD to unsupervised video denoising, where the lack of bias terms prevents the model from over-specializing to specific noise statistics.

2. Handling High Dynamic Range

In HDR imaging, light intensities can span several orders of magnitude. Chenyang Le and Kede Ma leverage bias-free architectures in Perceptually Optimized Deep High-Dynamic-Range Image Tone Mapping to create tone mappers that are scale-invariant. This allows a single model to process uncalibrated sensor data accurately, as the model's response scales proportionally with input luminance.

3. Generalization in Graphs and Physics

The benefits extend beyond pixels. Hao Tang introduced HomoGNN, which uses homogeneous layers to solve graph problems. By removing biases, a GNN trained on tiny graphs can solve shortest-path problems on graphs with thousands of nodes because it learns the logic of the path rather than the scale of the node attributes. Similarly, in NCLaw, Pingchuan Ma uses bias-free layers to ensure that a physical simulation produces zero stress when there is zero strain—a basic physical requirement that standard biased networks often violate.

Networks as Adaptive Linear Filters

A powerful way to visualize bias-free networks is as adaptive linear filters. As noted in Robustifying Fourier Features Embeddings for Implicit Neural Representations, a bias-free MLP can be written as $f(x) = M(x)x$, where $M(x)$ is a matrix that depends on the input.

In this view, the network isn't just a black box; it's a system that chooses a specific linear projection for every input. This makes the models highly interpretable. Robin Hesse and Stefan Roth show in Fast Axiomatic Attribution for Neural Networks that for these "$\mathcal{X}$-DNNs," feature attribution becomes mathematically trivial. Because of Euler's theorem for homogeneous functions, the input multiplied by the gradient exactly equals the output ($x \cdot \nabla f(x) = f(x)$), providing a perfect, closed-form explanation of the model's decision.

Conclusion

Removing bias terms is a rare example of a "less is more" architectural change. By sacrificing the ability to represent affine shifts, these networks gain a fundamental inductive bias—scale equivariance—that aligns with the natural symmetries of the physical world. Whether denoising a photon-starved microscopic image or solving a combinatorial problem, bias-free architectures ensure that the model's reasoning remains stable across the vast scales of real-world data.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scale_equivariance_through_bias_free_arc_394ea45b,
  title = {Scale Equivariance through Bias-Free Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-equivariance-through-bias-free-architectures/rdr_394ea45bca32415b8b863c855cc5d58b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_scale_equivariance_through_bias_free_arc_394ea45b_v1,
  title = {Scale Equivariance through Bias-Free Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-equivariance-through-bias-free-architectures/rdr_394ea45bca32415b8b863c855cc5d58b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}