Synthesize-It-Classifier: Learning a Generative Classifier through RecurrentSelf-analysis

Arghya Pal, Raphael C.-w. Phan, KokSheik Wong

Abstract

In this work, we show the generative capability of an image classifier network by synthesizing high-resolution, photo-realistic, and diverse images at scale. The overall methodology, called Synthesize-It-Classifier (STIC), does not require an explicit generator network to estimate the density of the data distribution and sample images from that, but instead uses the classifier's knowledge of the boundary to perform gradient ascent w.r.t. class logits and then synthesizes images using Gram Matrix Metropolis Adjusted Langevin Algorithm (GRMALA) by drawing on a blank canvas. During training, the classifier iteratively uses these synthesized images as fake samples and re-estimates the class boundary in a recurrent fashion to improve both the classification accuracy and quality of synthetic images. The STIC shows the mixing of the hard fake samples (i.e. those synthesized by the one hot class conditioning), and the soft fake samples (which are synthesized as a convex combination of classes, i.e. a mixup of classes) improves class interpolation. We demonstrate an Attentive-STIC network that shows an iterative drawing of synthesized images on the ImageNet dataset that has thousands of classes. In addition, we introduce the synthesis using a class conditional score classifier (Score-STIC) instead of a normal image classifier and show improved results on several real-world datasets, i.e. ImageNet, LSUN, and CIFAR 10.

Introduction

In deep learning, discriminative models (classifiers) and generative models (generators) are traditionally treated as distinct architectures. Classifiers excel at identifying decision boundaries but often lack an understanding of the underlying data distribution. Conversely, generative models like GANs can be difficult to train and may ignore the precise decision logic used by classifiers.

Previous attempts to bridge this gap, such as Joint Energy-Based Models, often struggled with boundary smoothness and tightness. These models frequently failed to distinguish between realistic data and noisy regions near the decision boundary, resulting in blurry or unrealistic images. The Synthesize-It-Classifier (STIC) framework addresses these challenges by transforming a standard classifier into a high-quality generator through a recurrent "self-analysis" loop.

Methodology

STIC treats the classifier itself as the generator. The core innovation lies in three primary components that allow the model to refine its own understanding of data classes:

  1. GRMALA (Gram Matrix Regularized Metropolis Adjusted Langevin Algorithm): Standard gradient ascent on class scores often results in adversarial noise. STIC incorporates a Gram Matrix regularizer—a technique from style transfer—to ensure synthesized images maintain realistic textures and structural regularities.
  2. Vicinal Risk Minimization (Mixup): By training on convex combinations of classes (e.g., a mix of "Cat" and "Dog"), the model learns smooth, continuous transitions, which is vital for generating realistic interpolations.
  3. Recurrent Self-Analysis: Training occurs in multiple passes (denoted by $\tau$). In each pass, the model synthesizes "fake" images based on its current state. In the subsequent pass, it uses these fakes as negative examples to tighten its class definitions.
The recurrent self-analysis loop and GRMALA sampling process.

Experimental Results

The STIC framework was evaluated on ImageNet, CIFAR-10, and LSUN datasets. The results demonstrate that the classifier's internal knowledge of complex objects—like cheeseburgers, mushrooms, and architectural structures—can be successfully extracted to create high-fidelity imagery.

Synthesized images from ImageNet, CIFAR-10, and LSUN (Church) datasets.

Quantitative and Qualitative Performance

  • State-of-the-Art Synthesis: STIC and its variant, Score-STIC, consistently outperformed existing hybrid models. On ImageNet, Score-STIC achieved a Fréchet Inception Distance (FID) of 8.83, significantly beating contemporary generative-discriminative hybrids.
  • Improved Classification: Unlike GANs, which often sacrifice classification utility for image quality, STIC actually improves the accuracy of the underlying classifier. A WideResNet trained with STIC saw its Top-5 accuracy on real images improve alongside its synthesis quality.
  • Efficiency: STIC trained approximately 3.2x faster than state-of-the-art GANs like BigGAN by avoiding the unstable "two-player game" dynamics of adversarial training.

Discussion and Future Work

While STIC offers significant advantages in training stability and classification accuracy, it faces limitations regarding sampling latency. Generating images at inference time requires multiple MCMC steps, which is slower than the single forward pass of a GAN generator. Additionally, the quality of synthesis is sensitive to the scaling factors of the GRMALA sampler.

Future research will explore "attentive" mechanisms to accelerate the sampling process in high-dimensional spaces and apply the STIC methodology to a broader range of discriminative architectures beyond standard ResNets.

Summary updated

Abstract

In this work, we show the generative capability of an image classifier network by synthesizing high-resolution, photo-realistic, and diverse images at scale. The overall methodology, called Synthesize-It-Classifier (STIC), does not require an explicit generator network to estimate the density of the data distribution and sample images from that, but instead uses the classifier's knowledge of the boundary to perform gradient ascent w.r.t. class logits and then synthesizes images using Gram Matrix Metropolis Adjusted Langevin Algorithm (GRMALA) by drawing on a blank canvas. During training, the classifier iteratively uses these synthesized images as fake samples and re-estimates the class boundary in a recurrent fashion to improve both the classification accuracy and quality of synthetic images. The STIC shows the mixing of the hard fake samples (i.e. those synthesized by the one hot class conditioning), and the soft fake samples (which are synthesized as a convex combination of classes, i.e. a mixup of classes) improves class interpolation. We demonstrate an Attentive-STIC network that shows an iterative drawing of synthesized images on the ImageNet dataset that has thousands of classes. In addition, we introduce the synthesis using a class conditional score classifier (Score-STIC) instead of a normal image classifier and show improved results on several real-world datasets, i.e. ImageNet, LSUN, and CIFAR 10.
BibTeX
@inproceedings{pal2021synthesize,
  author = {Arghya Pal and Raphael C.-W. Phan and KokSheik Wong},
  title = {{Synthesize-It-Classifier: Learning a Generative Classifier through RecurrentSelf-analysis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=2SMwZeQdER},
}
Figure