Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks
Lists
Only you see this
Abstract
By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using ideas from proximal point methods in convex analysis, we prove that this BCD algorithm will converge globally to a stationary point with R-linear convergence rate of order one. In experiments with the MNIST database, DNNs trained with this BCD algorithm consistently yielded better test-set error rates than identical DNN architectures trained via all the stochastic gradient descent (SGD) variants in the Caffe toolbox.
BibTeX
@inproceedings{zhang2017convergent,
author = {Ziming Zhang and Matthew Brand},
title = {{Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks}},
booktitle = {Advances in Neural Information Processing Systems},
year = {2017},
url = {https://openreview.net/forum?id=BJ-7C8Z_-B},
}Loading related content...