Mutual exclusivity loss for semi-supervised deep learning

Mehdi Sajjadi, Mehran Javanmardi, Tolga TasdizenICIP 2016 · DOI

TL;DR

This paper introduces a novel "mutual exclusivity loss" for semi-supervised deep learning that encourages a classifier's predictions for different classes to be mutually exclusive. This loss function pushes decision boundaries into low-density regions of the data space, leading to improved classification accuracy, especially when labeled data is scarce.

What Problem Does This Paper Address?

Training high-accuracy deep learning models, particularly Convolutional Neural Networks (ConvNets), typically requires vast amounts of labeled data. However, acquiring labeled data is expensive and time-consuming, while unlabeled data is abundant and cheap. Semi-supervised learning aims to leverage this readily available unlabeled data to improve model performance. Existing methods for semi-supervised learning have various approaches, including self-training, generative models, and graph-based methods. This paper addresses the challenge of effectively utilizing unlabeled data to enhance ConvNet performance by proposing a new regularization technique that explicitly guides the decision boundary to lie in sparse regions between class manifolds, thereby improving generalization.

Methods

The core contribution of this paper is the Mutual Exclusivity Loss. This is an unsupervised regularization term designed to be added to any backpropagation-based learning objective. The intuition behind the loss is to enforce that for any given input, the classifier's output probabilities for different classes are mutually exclusive. This is achieved by formulating a differentiable approximation of a logical "not" operation and a conjunction of binary variables.

Specifically, the authors define an ideal binary indicator function where only one element is 1 (representing the correct class) and others are 0. They then define a Boolean function I(f) that checks if a prediction f adheres to this mutual exclusivity. To make this differentiable and optimizable with gradient descent, they approximate the logical operations. The resulting unsupervised loss function, lu, penalizes predictions that are not mutually exclusive.

The total loss function for training becomes Itot = lc + λlu, where lc is the standard supervised loss (e.g., cross-entropy) and λ is a hyperparameter balancing the supervised and unsupervised terms. The authors argue that this loss implicitly pushes decision boundaries into low-density areas of the feature space, leading to better generalization.

Datasets

The authors evaluated their method on several standard computer vision datasets:

  • MNIST: A dataset of handwritten digits.
  • CIFAR10: A dataset of small color images in 10 classes.
  • NORB: A dataset of 3D objects viewed from different angles and lighting conditions.
  • SVHN (Street View House Numbers): A dataset of house numbers extracted from Google Street View images.
  • ILSVRC 2012 (ImageNet): A large-scale dataset with 1000 object categories.

Key Insights and Ideas

  • Mutual Exclusivity as a Regularizer: The paper proposes that forcing classifier outputs to be mutually exclusive for different classes is a powerful regularization technique for semi-supervised learning.
  • Decision Boundary Placement: The mutual exclusivity loss implicitly encourages decision boundaries to reside in low-density regions of the feature space, away from any training samples. This is a key principle for good generalization.
  • Effectiveness with Scarce Labels: The proposed method shows significant improvements when the amount of labeled data is very small, highlighting its utility in real-world scenarios where labeling is a bottleneck.
  • Generality: The loss function is designed to be compatible with any backpropagation-based learning method, making it broadly applicable.

Key Results

The paper demonstrates consistent improvements across various datasets when using the proposed mutual exclusivity loss, especially in low-label regimes:

  • MNIST: With only 0.13% labeled data, the error rate dropped from 20.96% to 14.82% compared to using labeled data only.
  • NORB: With 1% labeled data, the error rate improved from 26.77% to 19.02%.
  • SVHN: With 1% labeled data, the error rate decreased from 21.53% to 15.30%.
  • CIFAR10: With 1% labeled data, the error rate improved from 57.61% to 54.48%.
  • ImageNet: Using the AlexNet model, the proposed method achieved an error rate of 42.90%, compared to 45.63% without the regularization, even with a large number of classes.

In most experiments, increasing the amount of unlabeled data further improved performance. The authors also compared their method to entropy minimization regularization, showing their approach generally performed better, especially when labeled data was very limited.

Limitations and Future Work

  • Hyperparameter Tuning: The performance of the method is influenced by the hyperparameter λ, which balances the supervised and unsupervised losses. While the authors found it not overly sensitive in their experiments, optimal tuning might still be required.
  • Dataset Complexity: The authors note that simpler tasks (like MNIST) benefit more from unsupervised regularization than more challenging datasets (like CIFAR10). However, they suggest that with larger ConvNets and more unlabeled data, even complex tasks could see greater benefits.
  • Future Work: The paper implicitly suggests exploring larger ConvNet architectures and more extensive hyperparameter sweeps on more challenging datasets. Further investigation into the theoretical underpinnings of why mutual exclusivity leads to better decision boundary placement could also be a future direction.

Summary updated

BibTeX
@inproceedings{sajjadi2016mutual,
  author = {Mehdi Sajjadi and Mehran Javanmardi and Tolga Tasdizen},
  title = {{Mutual exclusivity loss for semi-supervised deep learning}},
  booktitle = {ICIP},
  year = {2016},
  doi = {10.1109/icip.2016.7532690},
}
Figure