Training Neural Networks with Implicit Variance

Justin Bayer, Christian Osendorfer, Sebastian Urban, Patrick van der Smagt

Abstract

We present a novel method to train predictive Gaussian distributions p(z|x) for regression problems with neural networks. While most approaches either ignore or explicitly model the variance as another response variable, it is trained implicitly in our case. Establishing stochasticty by the injection of noise into the input and hidden units, the outputs are approximated with a Gaussian distribution by the forward propagation method introduced for fast dropout [1]. We have designed our method to respect that probabilistic interpretation of the output units in the loss function. The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of likelihood.
BibTeX
@inproceedings{bayer2013training,
  author = {Justin Bayer and Christian Osendorfer and Sebastian Urban and Patrick van der Smagt},
  title = {{Training Neural Networks with Implicit Variance}},
  booktitle = {ICONIP},
  year = {2013},
  doi = {10.1007/978-3-642-42042-9\_17},
}

Loading related content...

Figure