Scoring and Classifying with Gated Auto-Encoders

Daniel Jiwoong Im, Graham W. Taylor

TL;DR

This paper introduces a novel way to "score" Gated Auto-Encoders (GAEs) by viewing them as dynamical systems, allowing them to be used for classification tasks. They demonstrate that GAEs, when scored, can outperform existing methods on both single and multi-label classification benchmarks.

What Problem Does This Paper Address?

Representation learning, particularly with deep learning, has seen tremendous success, but often relies on large amounts of labeled data. Unsupervised methods like auto-encoders are promising, but using them for classification has been challenging. Specifically, it was unclear how to assign a "score" (or how much a model "likes" an input) to data processed by an auto-encoder, and their non-probabilistic nature made direct calibration difficult. This paper tackles these issues by developing a scoring mechanism for Gated Auto-Encoders (GAEs), a more flexible variant of standard auto-encoders, and demonstrating its utility in classification.

Methods

The core contribution is applying a "dynamical systems view" to Gated Auto-Encoders (GAEs). This perspective treats the GAE's reconstruction process as a vector field. By leveraging a mathematical criterion (Poincaré's integrability criterion), the authors show that this vector field is conservative, meaning it can be derived from a scalar potential energy function. This energy function serves as the "score" for the GAE. The paper derives this scoring function for GAEs and shows its equivalence to the free energy of certain Restricted Boltzmann Machines (RBMs). This connection is crucial as RBMs have established theoretical grounding and have been used for classification.

Datasets

The paper evaluates its methods on a standard set of "Deep Learning Benchmarks" [11]. For classification tasks, they specifically mention using four multi-labeled datasets:

  • Yeast: Consists of biological attributes with 14 possible labels.
  • Scene: Image-based dataset with 6 possible labels.
  • MTurk: Audio features with 92 possible tags.
  • MajMin: Audio features with 96 possible tags.

Key Insights and Ideas

  • GAEs as Dynamical Systems: Viewing GAEs as dynamical systems provides a powerful theoretical lens, enabling the derivation of a scoring function. This moves beyond simply using GAEs for feature extraction.
  • Scoring for Classification: The derived scoring function allows GAEs to be used directly for classification, either by training class-specific GAEs or by combining their scores. This is a significant step towards leveraging unsupervised learning for supervised tasks.
  • RBM Equivalence: The theoretical connection between GAE scoring and the free energy of RBMs is a key insight. This bridges the gap between non-probabilistic and probabilistic models and suggests that GAEs can inherit some of the theoretical advantages of RBMs.
  • Multi-label Classification Potential: The paper demonstrates that GAE scoring is effective for the more complex task of multi-label classification, where multiple labels can be assigned to a single data point and correlations between labels exist.

Key Results

  • Single-label Classification: On the "Deep Learning Benchmarks," the proposed GAE scoring methods (cAES and mcAES) outperformed classical auto-encoder scoring (AES) and other baseline methods, achieving lower error rates. Specifically, mcAES (which models both mean and covariance) was generally the best performer.
  • Multi-label Classification: The proposed GAEXY and GAEy2 methods showed promising results on four multi-label datasets, generally outperforming logistic regression, a standard MLP, and other structured prediction algorithms. GAEXY achieved lower error rates on Yeast and MajMin, while GAEy2 performed better on Scene.
  • Efficiency: The paper highlights that inference using GAE scoring for structured output prediction is very fast, relying on gradient-based optimization.

Limitations and Future Work

  • Theoretical Study of GAEs: The paper notes that the theoretical study of GAEs is still limited, and their work aims to provide more insight.
  • Performance on Noise: The performance of GAEy2 was negatively affected by adding noise, suggesting that noise robustness might be an area for further investigation.
  • Future Work: The authors plan to tackle more challenging structured output prediction problems and explore further applications of their GAE scoring approach. They also mention that inference is extremely fast, suggesting this as an advantage for future work.

Summary updated

Abstract

Auto-encoders are perhaps the best-known non-probabilistic methods for representation learning. They are conceptually simple and easy to train. Recent theoretical work has shed light on their ability to capture manifold structure, and drawn connections to density modeling. This has motivated researchers to seek ways of auto-encoder scoring, which has furthered their use in classification. Gated auto-encoders (GAEs) are an interesting and flexible extension of auto-encoders which can learn transformations among different images or pixel covariances within images. However, they have been much less studied, theoretically or empirically. In this work, we apply a dynamical systems view to GAEs, deriving a scoring function, and drawing connections to Restricted Boltzmann Machines. On a set of deep learning benchmarks, we also demonstrate their effectiveness for single and multi-label classification.
BibTeX
@inproceedings{im2015scoring,
  author = {Daniel Jiwoong Im and Graham W. Taylor},
  title = {{Scoring and Classifying with Gated Auto-Encoders}},
  booktitle = {ECML/PKDD},
  year = {2015},
  url = {https://openreview.net/forum?id=VbLuBKCFNQ},
}
Figure