On The Implicit Bias of Weight Decay in Shallow Univariate ReLU Networks

Boris HaninICLR 2023 · OpenReview

Abstract

We give a complete characterization of the implicit bias of infinitesimal weight decay in the modest setting of univariate one layer ReLU networks. Our main result is a surprisingly simple geometric description of all one layer ReLU networks that exactly fit a dataset $\mathcal D= \set{(x_i,y_i)}$ with the minimum value of the $\ell_2$-norm of the neuron weights. Specifically, we prove that such functions must be either concave or convex between any two consecutive data sites $x_i$ and $x_{i+1}$. Our description implies that interpolating ReLU networks with weak $\ell_2$-regularization achieve the best possible generalization for learning $1d$ Lipschitz functions, up to universal constants.
BibTeX
@inproceedings{hanin2023implicit,
  author = {Boris Hanin},
  title = {{On The Implicit Bias of Weight Decay in Shallow Univariate ReLU Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=B8_T6-8-tCU},
}

Loading related content...

Figure