Unsupervised Panoptic Interpretation of Latent Spaces in GANs Using Space-Filling Vector Quantization

Mohammad Hassan Vali, Tom Bäckström

Introduction

Generative Adversarial Networks (GANs) have revolutionized image synthesis, producing high-fidelity visuals from low-dimensional latent spaces. However, these latent spaces often act as "black boxes," making it difficult to perform targeted edits—such as altering a subject's age or pose—without extensive manual supervision. Researchers have recently proposed a novel unsupervised approach called Space-Filling Vector Quantization (SFVQ) to "map" these spaces and discover interpretable transformation directions automatically.

The Challenge of Latent Space Navigation

Navigating a GAN's latent space typically requires identifying specific "directions" that correspond to semantic attributes. Existing methods generally fall into two categories:

  1. Supervised Methods: These rely on large labeled datasets or pre-trained classifiers to find attribute-specific vectors.
  2. Unsupervised Methods: Techniques like Principal Component Analysis (PCA) identify directions based on data variance. However, they are often constrained by mathematical requirements (e.g., orthogonality) and require users to predefined the number of directions ($K$).

The goal of this research is to provide a "panoptic" (all-encompassing) interpretation that is flexible, unsupervised, and intuitive for human users.

Methodology: Space-Filling Vector Quantization

The core innovation is the application of Space-Filling Vector Quantization (SFVQ). While standard Vector Quantization (VQ) represents a distribution using discrete points (codebook vectors), SFVQ arranges these points along a continuous, piece-wise linear curve.

Comparison between standard Vector Quantization (a) and the proposed Space-Filling Vector Quantization (b). SFVQ connects codebook vectors into a continuous path that maps the data distribution.

To adapt SFVQ for GAN latent spaces, the authors introduced two key technical refinements:

  • Smart Initialization: Latent vectors are sorted by their Euclidean norm before curve fitting, ensuring the initial path logically spans the distribution.
  • Improved Expansion: When increasing the number of points (codewords), the algorithm "shifts" existing points to maintain the curve's integrity within the data distribution, avoiding "outlier" segments that don't correspond to realistic images.

Key Insights

  • The Curve as a Global Map: The ordered nature of the SFVQ curve creates a universal map. By generating images along the curve, users can visually identify "neighborhoods" corresponding to specific features, such as age groups in faces or breeds in animal datasets.
  • Lines as Interpretable Directions: The individual segments of the curve serve as transformation directions. Unlike PCA-based methods, these directions are not forced to be perpendicular, allowing them to better capture "entangled" real-world features.
  • Discovery of Unique Attributes: The method successfully identified unique directions—such as "Hat" or "Beard" for human faces and "Bicolor" for animals—that were often missed by previous unsupervised techniques.

Experimental Results

The authors evaluated SFVQ on state-of-the-art models like StyleGAN2 and BigGAN across diverse datasets, including FFHQ (faces), AFHQ (animals), and LSUN Cars.

  • Human Preference: In subjective evaluations with 20 participants, SFVQ-derived directions were ranked highest for attributes like Smile, Hair Color, and Age, outperforming established baselines like GANSpace and SeFa.
  • Identity Preservation: Quantitative metrics showed that SFVQ preserves the original subject's identity better than competitors during transformations, particularly for complex edits like race or hair color.
  • Cross-Class Generalization: Concepts learned in one class (e.g., "Bicolor" in cats) were found to be applicable to others (e.g., dogs), indicating that the model captures fundamental visual concepts.

Conclusion

SFVQ provides a powerful, unsupervised tool for interpreting complex data distributions. While currently limited by the biases present in training datasets (e.g., the inability to find "bicolored lions" if they don't exist in the data), the method's class-agnostic nature makes it a promising candidate for interpreting other domains, such as video, speech, or scientific data distributions.

Summary updated

BibTeX
@article{vali2025unsupervised,
  author = {Mohammad Hassan Vali and Tom Bäckström},
  title = {{Unsupervised Panoptic Interpretation of Latent Spaces in GANs Using Space-Filling Vector Quantization}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=SEJatSGZX8},
}
Figure