Two-Layer Contractive Encodings with Shortcuts for Semi-supervised Learning

Hannes Schulz, KyungHyun Cho, Tapani Raiko, Sven Behnke

Abstract

Supervised training of multi-layer perceptrons (MLP) with only few labeled examples is prone to overfitting. Pretraining an MLP with unlabeled samples of the input distribution may achieve better generalization. Usually, pretraining is done in a layer-wise, greedy fashion which limits the complexity of the learnable features. To overcome this limitation, two-layer contractive encodings have been proposed recently—which pose a more difficult optimization problem, however. On the other hand, linear transformations of perceptrons have been proposed to make optimization of deep networks easier. In this paper, we propose to combine these two approaches. Experiments on handwritten digit recognition show the benefits of our combined approach to semi-supervised learning.
BibTeX
@inproceedings{schulz2013two,
  author = {Hannes Schulz and KyungHyun Cho and Tapani Raiko and Sven Behnke},
  title = {{Two-Layer Contractive Encodings with Shortcuts for Semi-supervised Learning}},
  booktitle = {ICONIP},
  year = {2013},
  doi = {10.1007/978-3-642-42054-2\_56},
}

Loading related content...

Figure