Introspective Classification with Convolutional Nets

Long Jin, Justin Lazarow, Zhuowen Tu

TL;DR

This paper introduces Introspective Convolutional Networks (ICN), a framework that allows a single CNN to act as both a classifier and a generator. By synthesizing its own "hard negative" samples and iteratively retraining on them, the model refines its decision boundaries, leading to superior classification performance and increased robustness against adversarial attacks.

What Problem Does This Paper Address?

Despite their massive success, deep convolutional neural networks (CNNs) often make "absurd" mistakes—classifying noise or slightly perturbed images with high confidence in ways that a human never would. Traditionally, researchers have tried to fix this by providing more data or using data augmentation (like flipping or cropping images).

However, these methods rely on external data or human-defined transformations. The authors of this paper ask: Can a model improve itself by looking inward? They aim to bridge the gap between discriminative models (which predict labels) and generative models (which create data) within a single architecture, using the model's own internal "understanding" to generate the very examples it finds most confusing.

Methods

The core of the paper is an iterative algorithm called reclassification-by-synthesis.

Reclassification-by-synthesis process

It consists of two main steps:

  1. Synthesis Step: The model generates "pseudo-negative" samples. Starting from random noise, the algorithm uses stochastic gradient descent (SGD) or Langevin dynamics to "evolve" the noise into something the current classifier thinks is a positive example. Essentially, it performs backpropagation with respect to the input image rather than the weights.
  2. Reclassification Step: These newly synthesized images are labeled as "negatives" and added to the training set. The CNN is then retrained on this augmented dataset.

Mathematically, this is grounded in Bayes' Theorem. The authors show that a discriminative classifier can be transformed into a generative model if the model can effectively sample from the distribution of "confusing" negatives. Unlike Generative Adversarial Networks (GANs), which use two separate networks (a generator and a discriminator) playing a minimax game, ICN uses a single model that performs both tasks.

Datasets

The authors evaluated ICN on three standard computer vision benchmarks:

  • MNIST: 70,000 grayscale images of handwritten digits.
  • CIFAR-10: 60,000 low-resolution color images across 10 categories (airplane, bird, etc.).
  • SVHN (Street View House Numbers): Over 600,000 images of house numbers cropped from Google Street View.

They used state-of-the-art architectures like ResNet-32 as the base for their experiments to demonstrate that ICN provides a boost even to already powerful models.

Key Insights and Ideas

  • The Discriminator is the Generator: The most elegant idea here is that you don't need a separate "Generator" network. The features learned for classification are sufficient to guide the synthesis of new data.
  • Tightening the Boundary: In early training stages, the synthesized samples look like random noise. As the model improves, these "pseudo-negatives" begin to look more like real objects.
Synthesized MNIST digits over training iterations

This forces the model to learn much finer, more precise decision boundaries that distinguish real data from "almost-real" noise.

  • Efficiency via Backprop: By using backpropagation for sampling (similar to techniques used in DeepDream), the authors made the generative process much faster than older Markov Chain Monte Carlo (MCMC) methods, making it practical for deep networks.
  • Adversarial Robustness: Because the model is constantly "attacking" itself during training by generating confusing samples, it naturally becomes more resistant to external adversarial attacks.

Key Results

The ICN approach consistently outperformed baseline CNNs across all datasets:

  • MNIST: Reduced the test error from 0.77% (baseline) to 0.72%.
  • CIFAR-10: On a ResNet-32 backbone with data augmentation, the error dropped from 7.06% to 6.70%.
  • SVHN: Improved the error rate from 2.01% to 1.95%.
  • Robustness: In a head-to-head test against adversarial examples, the ICN classifier was significantly harder to "fool" than a standard CNN, showing a 38.7% reduction in error against adversarial attacks.

Limitations and Future Work

  • Training Overhead: The primary drawback is computational cost. Training an ICN takes roughly 1.7 to 2.1 times longer than a standard CNN because of the iterative synthesis steps.
  • Sample Quality: While the synthesized samples are excellent for training the classifier, they are not necessarily "photorealistic" in the way that modern GAN or Diffusion model outputs are. The goal here is discriminative power, not aesthetic generation.
  • Scaling: While tested on CIFAR and SVHN, scaling this "self-synthesis" approach to high-resolution datasets like ImageNet remains a challenge due to the increased cost of the synthesis step at higher resolutions.

The authors suggest that future work could explore more sophisticated sampling strategies and investigate how this "introspective" approach might benefit unsupervised or semi-supervised learning tasks.

Summary updated

Abstract

We propose introspective convolutional networks (ICN) that emphasize the importance of having convolutional neural networks empowered with generative capabilities. We employ a reclassification-by-synthesis algorithm to perform training using a formulation stemmed from the Bayes theory. Our ICN tries to iteratively: (1) synthesize pseudo-negative samples; and (2) enhance itself by improving the classification. The single CNN classifier learned is at the same time generative --- being able to directly synthesize new samples within its own discriminative model. We conduct experiments on benchmark datasets including MNIST, CIFAR-10, and SVHN using state-of-the-art CNN architectures, and observe improved classification results.
BibTeX
@inproceedings{jin2017introspective,
  author = {Long Jin and Justin Lazarow and Zhuowen Tu},
  title = {{Introspective Classification with Convolutional Nets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=rkWEmDZdZB},
}
Figure