Abstract
Within the framework of generative adversarial networks (GANs), we propose objectives that task the discriminator for self-supervised representation learning via additional structural modeling responsibilities. In combination with an efficient smoothness regularizer imposed on the network, these objectives guide the discriminator to learn to extract informative representations, while maintaining a generator capable of sampling from the domain. Specifically, our objectives encourage the discriminator to structure features at two levels of granularity: aligning distribution characteristics, such as mean and variance, at coarse scales, and grouping features into local clusters at finer scales. Operating as a feature learner within the GAN framework frees our self-supervised system from the reliance on hand-crafted data augmentation schemes that are prevalent across contrastive representation learning methods. Across CIFAR-10/100 and an ImageNet subset, experiments demonstrate that equipping GANs with our self-supervised objectives suffices to produce discriminators which, evaluated in terms of representation learning, compete with networks trained by contrastive learning approaches.
Introduction
Self-supervised learning (SSL) has revolutionized computer vision by enabling models to learn semantic features from unlabeled data. Currently, the dominant paradigm is contrastive learning, which relies heavily on identifying "augmented views" of the same image. While effective, this approach is tethered to human-designed augmentation heuristics (e.g., specific crops, color jitters). If these augmentations do not align with the underlying invariances of a specific domain, the learned representations suffer.
Generative Adversarial Networks (GANs) present a theoretically "pure" alternative. In a GAN, the discriminator must learn to distinguish real data from synthetic samples, a task that should inherently require understanding data structure. However, standard GAN discriminators often fail as general-purpose feature learners because they prioritize detecting low-level artifacts to "win" against the generator. This research introduces "structural" objectives that transform the discriminator into a robust self-supervised learner without requiring complex augmentation pipelines.
Methodology: Structural Adversarial Objectives
The core innovation lies in moving beyond the standard scalar output of a discriminator. Instead of a single "real vs. fake" score, the discriminator is tasked with modeling the data manifold at multiple scales.
Coarse-scale and Fine-grained Optimization
At a global level, the discriminator is required to align the statistical characteristics of real and generated data. By producing feature vectors rather than scalars, the model can use Jensen-Shannon Divergence (JSD) to match the mean and variance of the "real" feature distribution with the "generated" one. This Distribution-level alignment forces the discriminator to capture the broad geometry of the data space.
To capture semantic nuances, the authors implement an Instance-level clustering objective. Using a mean-shift approach, the discriminator is encouraged to group similar samples together in the embedding space. This pressure to cluster encourages the model to discover latent categories (e.g., grouping various instances of "cars" or "birds") to better organize its internal representations.
Smoothness via Jacobian Spectral Normalization
To stabilize training without sacrificing model capacity, the authors propose a novel Jacobian-based smoothness regularizer. Unlike standard Spectral Normalization, which constrains weights layer-by-layer, this method approximates the spectral norm of the Jacobian matrix (the first-order partial derivatives of the output with respect to the input). This ensures the discriminator's learned manifold is smooth and well-behaved, which is critical for both stable GAN training and high-quality feature extraction.
Experimental Results
The proposed framework was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and subsets of ImageNet.
Representation Learning Performance
The structural objectives significantly improve the discriminator's utility as a feature learner. On CIFAR-10, the method achieved a 75.4% accuracy using K-Means clustering on the learned features, notably outperforming SimCLR (65.7%) when evaluated in an augmentation-free setting. When using linear probing (training a linear classifier on top of frozen features), the model reached 89.8% accuracy, placing it on par with state-of-the-art contrastive methods like BYOL and DINO.
Impact on Generative Quality
Interestingly, the structural tasks do not just benefit the discriminator; they also improve the generator. By learning more semantically meaningful features, the discriminator provides a more informative gradient to the generator. This resulted in a lower (better) Fréchet Inception Distance (FID) score of 13.63 on CIFAR-10, compared to 18.54 for a standard Hinge Loss GAN.
Conclusion
This work demonstrates that the GAN framework is a viable and powerful alternative to contrastive learning for self-supervised representation learning. By imposing structural modeling responsibilities—specifically distribution alignment and local clustering—the discriminator learns deep semantic features that are competitive with state-of-the-art methods. Crucially, this is achieved without the need for hand-crafted data augmentations, suggesting a more generalizable path for SSL in domains where appropriate augmentations are not intuitively obvious.
Summary updated