A Two-Layer ICA-Like Model Estimated by Score Matching

Urs Köster, Aapo Hyvärinen

Abstract

Capturing regularities in high-dimensional data is an important problem in machine learning and signal processing. Here we present a statistical model that learns a nonlinear representation from the data that reflects abstract, invariant properties of the signal without making requirements about the kind of signal that can be processed. The model has a hierarchy of two layers, with the first layer broadly corresponding to Independent Component Analysis (ICA) and a second layer to represent higher order structure. We estimate the model using the mathematical framework of Score Matching (SM), a novel method for the estimation of non-normalized statistical models. The model incorporates a squaring nonlinearity, which we propose to be suitable for forming a higher-order code of invariances. Additionally the squaring can be viewed as modelling subspaces to capture residual dependencies, which linear models cannot capture.
BibTeX
@inproceedings{koster2007two,
  author = {Urs Köster and Aapo Hyvärinen},
  title = {{A Two-Layer ICA-Like Model Estimated by Score Matching}},
  booktitle = {ICANN},
  year = {2007},
  doi = {10.1007/978-3-540-74695-9\_82},
}

Loading related content...

Figure