On the Learnability of Random Deep Networks
Lists
Only you see this
Abstract
In this paper we study the learnability of random deep networks both theoretically and experimentally. On the theoretical front, assuming the statistical query model, we show that the learnability of random deep networks with sign activation drops exponentially with their depths; under plausible conjectures, our results extend to ReLu and sigmoid activations. The core of the arguments is that even for highly correlated inputs, the outputs of deep random networks are near-orthogonal. On the experimental side, we find that the learnability of random networks drops sharply with depth even with the state-of-the-art training methods.
BibTeX
@inproceedings{das2020learnability,
author = {Abhimanyu Das and Sreenivas Gollapudi and Ravi Kumar and Rina Panigrahy},
title = {{On the Learnability of Random Deep Networks}},
booktitle = {SODA},
year = {2020},
url = {https://openreview.net/forum?id=ong-9lU4cGcn},
}Loading related content...