On rectified linear units for speech processing

Matthew D. Zeiler, Marc'Aurelio Ranzato, Rajat Monga, Mark Z. Mao, K. Yang, Quoc Viet Le, Patrick Nguyen, Andrew W. Senior, Vincent Vanhoucke, Jeffrey Dean, Geoffrey E. HintonICASSP 2013 · OpenReview

Abstract

Deep neural networks have recently become the gold standard for acoustic modeling in speech recognition systems. The key computational unit of a deep network is a linear projection followed by a point-wise non-linearity, which is typically a logistic function. In this work, we show that we can improve generalization and make training of deep networks faster and simpler by substituting the logistic units with rectified linear units. These units are linear when their input is positive and zero otherwise. In a supervised setting, we can successfully train very deep nets from random initialization on a large vocabulary speech recognition task achieving lower word error rates than using a logistic network with the same topology. Similarly in an unsupervised setting, we show how we can learn sparse features that can be useful for discriminative tasks. All our experiments are executed in a distributed environment using several hundred machines and several hundred hours of speech data.
BibTeX
@inproceedings{zeiler2013rectified,
  author = {Matthew D. Zeiler and Marc'Aurelio Ranzato and Rajat Monga and Mark Z. Mao and K. Yang and Quoc Viet Le and Patrick Nguyen and Andrew W. Senior and Vincent Vanhoucke and Jeffrey Dean and Geoffrey E. Hinton},
  title = {{On rectified linear units for speech processing}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=okJNhnBZQNM},
}

Loading related content...

Figure