Deep learning via semi-supervised embedding

Jason Weston, Frédéric Ratle, Ronan CollobertICML 2008 · OpenReview

TL;DR

This paper introduces a straightforward way to boost deep learning performance using unlabeled data by incorporating nonlinear embedding as a regularizer. By training a network to simultaneously classify labeled data and preserve the "manifold structure" of unlabeled data, the authors achieve state-of-the-art results on tasks like image recognition and natural language processing—without the complexity of traditional generative pre-training.

The Challenge of Deep Semi-Supervised Learning

Deep learning thrives on massive amounts of data, but labeled data is notoriously expensive and scarce. Semi-supervised learning (SSL) aims to bridge this gap by using easily obtainable unlabeled data to guide the learning process.

At the time of this paper's publication, most SSL techniques were "shallow" (like LapSVM), meaning they couldn't learn complex, hierarchical features. Conversely, existing "deep" SSL methods—such as Deep Belief Networks (DBNs) or stacked autoencoders—were often cumbersome. They typically required a two-stage process: first, a greedy, layer-by-layer unsupervised pre-training phase, followed by supervised fine-tuning. The authors sought a simpler, unified framework where supervised and unsupervised tasks are trained simultaneously within a deep architecture.

Methods: Introducing EmbedNN

The authors propose EmbedNN, a method that treats unsupervised learning as an auxiliary task. The core idea is to use a "Siamese-style" loss function that encourages the network to map similar inputs to nearby points in the embedding space.

They suggest three primary ways to "plug in" this embedding regularizer into a neural network:

  1. Output Embedding: Apply the regularizer directly to the final output layer.
  2. Internal Embedding: Apply it to a specific hidden layer.
  3. Auxiliary Embedding: Create a separate "branch" of the network that shares early layers with the classifier but has its own weights dedicated to the embedding task.
Three architectures for semi-supervised embedding: (a) Output, (b) Internal, and (c) Auxiliary.

The training process uses Stochastic Gradient Descent (SGD). For every labeled example used to minimize classification error, the algorithm picks a pair of unlabeled "neighbors" (points that are close in the original input space) and updates the weights to ensure their internal representations stay close. It also picks "non-neighbors" and ensures they are pushed apart to avoid the "collapse" problem where the network maps everything to a single point.

Key Insights and Ideas

  • Joint Training vs. Pre-training: Unlike DBNs, which train layers one by one, EmbedNN trains everything at once. This simplifies the pipeline and allows the supervised signal to influence the feature extraction from the very start.
  • Embedding as Regularization: The unsupervised task acts as a powerful "structural" regularizer. In deep networks, which are highly prone to overfitting, forcing the hidden layers to respect the geometry of the data manifold prevents the weights from becoming too specialized to the small labeled set.
  • The Power of Neighbors: The authors highlight that "neighbor" information doesn't always require a complex $k$-nearest neighbor search. In video, consecutive frames are natural neighbors; in text, words in similar contexts are neighbors. This makes the approach highly scalable to different domains.

Key Results

The study evaluates the approach across a diverse set of benchmarks, including small-scale SSL datasets (g50c, Text, Uspst), MNIST, and a massive Semantic Role Labeling (SRL) task.

  • MNIST: On the "Mnist1h" task (using only 100 labeled examples), a standard neural network had a 25.81% error rate. Adding the embedding regularizer dropped that error significantly to 17.05%.
  • Depth Matters: Shallow SSL methods perform well on small datasets but eventually plateau. Deep architectures, when regularized with embedding, continue to improve as they get deeper. In a 15-layer network, a standard model reached a disastrous 47.7% error due to overfitting and optimization issues. The EmbedNN version achieved a 9.3% error, proving that embedding losses make training very deep networks much more stable.
Error rates across varying network depths, demonstrating that embedding regularizers prevent catastrophic overfitting in very deep networks.
  • NLP Breakthrough: On the Semantic Role Labeling task, the EmbedNN approach (using 631 million unlabeled words from Wikipedia) achieved a 14.55% error rate. This outperformed state-of-the-art systems of the time (like ASSERT) that relied heavily on hand-coded linguistic features and parse trees.

Limitations and Future Work

The primary bottleneck of this approach is the construction of the similarity matrix $W$ (deciding which data points are neighbors). For massive datasets, calculating $k$-nearest neighbors is computationally expensive ($O(N^2)$). While the authors suggest sampling and using domain-specific heuristics (like temporal proximity in video), finding a more general, efficient solution for identifying neighbors in high-dimensional space remains an open challenge.

Additionally, the paper focuses on a specific margin-based loss for embedding. Exploring how other manifold learning techniques—such as t-SNE or Large Margin Nearest Neighbor—might fit into this deep, joint-training framework is suggested as a promising direction for future research.

Summary updated

Abstract

We show how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture. This provides a simple alternative to existing approaches to deep learning whilst yielding competitive error rates compared to those methods, and existing shallow semi-supervised techniques.
BibTeX
@inproceedings{weston2008deep,
  author = {Jason Weston and Frédéric Ratle and Ronan Collobert},
  title = {{Deep learning via semi-supervised embedding}},
  booktitle = {International Conference on Machine Learning},
  year = {2008},
  url = {https://openreview.net/forum?id=rk-YOhZd-r},
}

Loading related content...

Figure