Abstract
In this work, we show the generative capability of an image classifier network by synthesizing high-resolution, photo-realistic, and diverse images at scale. The overall methodology, called Synthesize-It-Classifier (STIC), does not require an explicit generator network to estimate the density of the data distribution and sample images from that, but instead uses the classifier's knowledge of the boundary to perform gradient ascent w.r.t. class logits and then synthesizes images using Gram Matrix Metropolis Adjusted Langevin Algorithm (GRMALA) by drawing on a blank canvas. During training, the classifier iteratively uses these synthesized images as fake samples and re-estimates the class boundary in a recurrent fashion to improve both the classification accuracy and quality of synthetic images. The STIC shows that mixing of the hard fake samples (i.e. those synthesized by the one hot class conditioning), and the soft fake samples (which are synthesized as a convex combination of classes, i.e. a mixup of classes) improves class interpolation. We demonstrate an Attentive-STIC network that shows iterative drawing of synthesized images on the ImageNet dataset that has thousands of classes. In addition, we introduce the synthesis using a class conditional score classifier (Score-STIC) instead of a normal image classifier and show improved results on several real world datasets, i.e. ImageNet, LSUN and CIFAR 10.
Introduction
Traditionally, machine learning has been divided into two distinct paradigms: discriminative models, which predict a label $y$ given an image $x$, and generative models, which learn the distribution of $x$ to create new data. While hybrid models have attempted to bridge this gap, they often struggle to produce high-resolution, photo-realistic images. Standard classifiers are frequently "overconfident" and possess jagged, poorly defined decision boundaries. When these models are inverted to synthesize images, the results are typically noisy and uninterpretable.
Synthesize-It-Classifier (STIC) addresses this by turning a standard image classifier into a high-quality image generator without the need for a separate GAN-style generator network. STIC demonstrates that a classifier's understanding of class boundaries, if sufficiently smooth and detailed, can be leveraged to "draw" realistic examples from scratch.
Methodology
The STIC framework relies on three primary innovations to enable high-fidelity synthesis and robust classification:
1. GRMALA Sampler
Standard sampling methods like the Metropolis Adjusted Langevin Algorithm (MALA) often produce noisy results or converge slowly. STIC introduces GRMALA (Gram Matrix Regularized MALA), which incorporates Gram Matrices—a technique borrowed from style transfer—to regularize the gradient ascent process. This forces synthesized images to maintain structural and textural consistency, ensuring that natural textures like fur or food appear realistic.
2. Recurrent Self-Analysis
The model is trained through an iterative "recurrent self-analysis" loop. In each pass $\tau$, the classifier learns from real data and then synthesizes "fake" images based on its current state. In the subsequent pass $\tau+1$, these synthesized images are used as negative samples to tighten the class boundaries. This self-correction mechanism forces the model to refine its internal representation of each class.
3. Smoothness via Mixup
To prevent "dead zones" between classes and enable smooth interpolation, STIC utilizes Vicinal Risk Minimization (Mixup). By training on convex combinations of images and labels, the model learns to generate meaningful images even when morphing between two distinct classes (e.g., a cat and a dog).
Experiments and Results
The authors evaluated STIC on three major benchmarks: ImageNet (1,000 classes), CIFAR-10, and LSUN (Church Class).
Quantitative Performance
STIC significantly outperformed previous generative-discriminative hybrids. On the CIFAR-10 dataset, the Score-STIC variant achieved a Frechet Inception Distance (FID) of 9.99, a substantial improvement over the 47.60 achieved by JEM. Furthermore, the recurrent training improved classification robustness; classifiers trained with STIC showed higher accuracy when tested on synthetic images ($Cls_G$).
Qualitative Quality and Efficiency
Qualitatively, the model produced recognizable, high-resolution images of complex ImageNet classes, such as cheeseburgers and mushrooms. Notably, the training process was approximately 3.2x faster than state-of-the-art GANs like BigGAN or SNGAN, as it avoids the unstable adversarial "arms race" between separate generator and discriminator networks.
Conclusion
STIC proves that explicit generator networks are not strictly necessary for high-quality image synthesis. By treating a classifier's own "hallucinations" as training data, the model performs a form of cognitive self-analysis that leads to superior decision boundaries and generative capabilities. While MCMC-based sampling remains slower than a single GAN forward pass, the efficiency gains in training and the quality of the resulting classifier-generator hybrid mark a significant step forward in unified machine learning models.
Summary updated