Towards Principled Unsupervised Learning

Ilya Sutskever, Rafal Józefowicz, Karol Gregor, Danilo Jimenez Rezende, Timothy P. Lillicrap, Oriol VinyalsarXiv 2015 · arXiv

This 2016 contribution from Google Brain and DeepMind addresses a fundamental challenge in machine learning: bridging the gap between unsupervised objectives and supervised performance. While supervised learning directly optimizes for the task at hand, unsupervised methods often focus on reconstruction or density modeling, which may not capture the features necessary for downstream classification.

The Problem: Misaligned Objectives

Traditional unsupervised learning focuses on capturing the input distribution $P(x)$. However, methods like autoencoders or density modeling have no inherent knowledge of the supervised task $P(y|x)$. A model might become highly proficient at reconstructing pixels while failing to learn the semantic features required for classification.

For unsupervised learning to be truly effective, the authors argue that the objective must be consistent with the supervised task. They propose that a principled approach should utilize unlabeled inputs and unlabeled outputs (the known distribution of labels without corresponding inputs) to guide the model toward the correct mapping.

Comparison of supervised cost alignment with typical unsupervised cost versus ODM cost.

Methods: Output Distribution Matching (ODM)

The core proposal is the Output Distribution Matching (ODM) cost. The intuition is simple: if a model $F$ correctly maps inputs $x$ to labels $y$, then the distribution of the model's predictions $F(x)$ must match the known distribution of the labels $y$.

The paper explores several architectures to optimize this:

  1. The $xyh$-Model & Dual Autoencoder: The authors conceptualize a generative model where a latent variable $h$ generates both the input $x$ and the label $y$. This is implemented via a "Dual Autoencoder" where two networks—one for the input domain and one for the output domain—share their innermost weights, forcing a shared representation.
  2. Generative Adversarial Networks (GANs): To match distributions in complex spaces, such as sequences of text, the authors employ a GAN framework. Here, a classifier acts as the generator, while a discriminator ensures the predictions match the statistics (e.g., n-grams) of the target language.
  3. One-Shot Domain Adaptation: The framework is used to classify test points from different distributions (e.g., inverted colors) by finding the transformation that allows the point to "fit" the training distribution.
The xyh-model and Dual Autoencoder architectures for shared representation learning.

Key Insights

  • Consistency is Key: Unlike reconstruction costs, the ODM cost is consistent because a perfect supervised classifier is, by definition, a global minimum of the ODM objective.
  • The Power of Large Output Spaces: ODM becomes more informative as the output space grows. In large spaces like text sequences, the distribution matching constraint is much harder to satisfy with an incorrect function than in simple binary classification.
  • Capacity Control: Experiments with the Dual Autoencoder revealed that Sigmoid units outperformed ReLUs. The authors suggest that the slower learning and inherent capacity constraints of Sigmoids prevented the model from overfitting to noise before finding the correct correspondence.

Experimental Results

  • MNIST Permutation: The Dual Autoencoder successfully recovered pixel permutation mappings without any labeled examples, effectively "deciphering" the image structure.
  • Extreme Semi-supervised Learning: Using GAN-based ODM, the model achieved a 4.7% test error on MNIST using only 4 labeled examples. However, performance dropped sharply with only 2 labels (72% error), indicating a threshold for stability.
  • Cipher Solving: The model perfectly identified symbol mappings in character and word ciphers in under 10,000 updates, relying solely on linguistic statistics.
  • One-Shot Adaptation: The model successfully classified inverted MNIST digits by inferring the necessary transformation on the fly, despite never seeing inverted images during training.

Limitations and Future Work

While promising, the authors note that GAN training for ODM remains fragile and sensitive to hyperparameters. Furthermore, the method relies on the assumption that input and output domains share an underlying hidden structure. Scaling these principles to more complex tasks, such as full-scale machine translation or attention-based models, remains a significant open challenge.

Summary updated

Abstract

General unsupervised learning is a long-standing conceptual problem in machine learning. Supervised learning is successful because it can be solved by the minimization of the training error cost function. Unsupervised learning is not as successful, because the unsupervised objective may be unrelated to the supervised task of interest. For an example, density modelling and reconstruction have often been used for unsupervised learning, but they did not produced the sought-after performance gains, because they have no knowledge of the supervised tasks. In this paper, we present an unsupervised cost function which we name the Output Distribution Matching (ODM) cost, which measures a divergence between the distribution of predictions and distributions of labels. The ODM cost is appealing because it is consistent with the supervised cost in the following sense: a perfect supervised classifier is also perfect according to the ODM cost. Therefore, by aggressively optimizing the ODM cost, we are almost guaranteed to improve our supervised performance whenever the space of possible predictions is exponentially large. We demonstrate that the ODM cost works well on number of small and semi-artificial datasets using no (or almost no) labelled training cases. Finally, we show that the ODM cost can be used for one-shot domain adaptation, which allows the model to classify inputs that differ from the input distribution in significant ways without the need for prior exposure to the new domain.
BibTeX
@misc{sutskever2015towards,
  author = {Ilya Sutskever and Rafal Józefowicz and Karol Gregor and Danilo Jimenez Rezende and Timothy P. Lillicrap and Oriol Vinyals},
  title = {{Towards Principled Unsupervised Learning}},
  year = {2015},
  eprint = {1511.06440},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.06440},
}
Figure