This paper explores a specific but powerful regime of machine learning: training only the biases of a neural network while keeping the weights fixed at their random initialization. While standard theory often focuses on training all parameters, bias-only training is increasingly relevant for efficient fine-tuning of large models.
TL;DR
Training only the biases of a ReLU network is equivalent to performing linear least squares on "activation codes"—binary vectors representing which neurons are active for a given input. As networks grow wider, these codes become increasingly stable, causing the training dynamics to converge to a predictable kernel regression governed by the Hamming distance between inputs.
What Problem Does This Paper Address?
In the "infinite-width" limit, neural network training is often described by the Neural Tangent Kernel (NTK), which simplifies complex gradient descent into a linear kernel regression. However, most NTK research assumes all weights and biases are trained together.
Recent empirical work has shown that biases alone carry surprising expressive power; you can actually achieve universal approximation just by shifting the activation thresholds (biases) of a fixed random network. The gap this paper fills is a theoretical understanding of the dynamics of this process. Specifically, how does the discrete, "piecewise" nature of ReLU activations interact with the continuous process of gradient descent when only biases are moving?
Methods
The authors focus on one-hidden-layer ReLU networks. Their key analytical tool is the activation code: a binary vector $z(x)$ where each entry is $1$ if a neuron is active for input $x$ and $0$ otherwise.
- Geometric Interpretation: They show that the input space is partitioned into convex polyhedral regions, each defined by a unique activation code. Training the bias effectively "slides" these polyhedral boundaries.
- Kernel Derivation: They prove that the bias-only NTK is exactly the inner product of these activation codes. This means the network's "similarity" metric is purely combinatorial.
- Lazy Training Analysis: They investigate the "lazy training" regime, where parameters stay close to their initialization. They bound the number of "neuron flips" (neurons changing from active to inactive or vice versa) to show that as the network gets wider, these flips become negligible.
Datasets
The authors use synthetic datasets to precisely control and measure the variables predicted by their theory:
- Inputs: 30 training points and 500 test points sampled uniformly from a unit sphere in $\mathbb{R}^5$.
- Targets: A smooth non-linear function $y = \sin(2x_1) + \frac{1}{2}x_2^2$.
- Perturbations: They created "nearby" pairs of test points (distance $10^{-3}$ to $10^{-1}$) to specifically test how the Hamming distance between codes affects training similarity.
Key Insights and Ideas
- Training as Sequence of Least Squares: As long as no neurons "flip" their activation state, bias-only training is mathematically identical to linear least squares using the activation codes as features. Training is thus a sequence of these problems, with transitions occurring only when a boundary is crossed.
- Hamming Distance is the Metric: The similarity between how two inputs are treated during training is directly proportional to their Hamming distance (the number of bits that differ in their activation codes). If two inputs share the same code, they receive identical updates.
- The "Frozen" Limit: In the infinite-width limit, the fraction of neurons that flip during training vanishes at a rate of $O(m^{-1/2})$. This means that for very wide networks, you can essentially treat the activation codes as fixed from the start.
- Arc-Cosine Connection: At initialization, the bias-only kernel converges to a scaled version of the arc-cosine kernel, a well-known kernel in deep learning theory.
Key Results
The paper provides both theoretical proofs and experimental validation:
- Kernel Stability: Experiments confirmed that the "kernel drift" (how much the NTK changes during training) shrinks as $O(m^{-1/2})$, matching the theoretical bound.
- Neuron Flips: While the absolute number of neurons that change state increases with width (at a rate of $\sqrt{m}$), their fraction relative to the total number of neurons decreases, as shown in the log-log plots of Figure 2(c).
- Prediction Similarity: Figure 2(b) demonstrates that the difference in how two inputs evolve is strictly bounded by the Hamming distance of their codes, validating the geometric intuition.
Limitations and Future Work
- Single Hidden Layer: The current analysis is strictly for one-hidden-layer networks. Deep networks involve compositions of these activation codes, which significantly complicates the geometry.
- ReLU Specificity: The results rely heavily on the piecewise-linear nature of ReLU. Other activations (like Sigmoid or GeLU) would not yield the same discrete "activation code" structure.
- Future Directions: The authors suggest extending this framework to multi-layer architectures to see if the "combinatorial" nature of the bias-only NTK persists in deeper models.
Summary updated