Plug-in Factorization for Latent Representation Disentanglement

Jee Seok Yoon, Wonjun Ko, Heung-Il SukarXiv 2019 · arXiv

Introduction

Deep learning models, particularly Generative Adversarial Networks (GANs) and Autoencoders, typically learn "latent representations"—compact numerical codes representing data. However, these codes are often entangled, meaning a single value might influence multiple semantic features simultaneously (e.g., age, gender, and pose). While disentangled representations are highly desirable for interpretability and control, most existing methods require training models from scratch. This paper introduces the Factors Decomposer-Entangler Network (FDEN), a "plug-in" module designed to factorize existing, entangled latent spaces into independent, interpretable components without modifying the original model's weights.

The FDEN Framework

The FDEN architecture functions as an autoencoder for latent vectors, comprising three primary modules:

  1. Decomposer ($\mathcal{D}$): This module maps the original latent vector $z$ into a set of discrete factors $f_0, f_1, ..., f_N$.
  2. Factorizer ($\mathcal{F}$): Serving as the core of the disentanglement process, the Factorizer employs a Statisticians Network to minimize Total Correlation (a multi-variable extension of mutual information) between factors, ensuring statistical independence. An Alignment Network provides semi-supervised guidance, linking specific factors to human-understandable attributes using minimal labeled data.
  3. Entangler ($\mathcal{E}$): This module recombines the factors to reconstruct the original latent vector $\tilde{z}$.
Figure 1: Detailed architecture of the FDEN, illustrating the Decomposer, Factorizer (with GRL and Statisticians/Alignment networks), and Entangler modules.

To ensure stable training, the authors utilize the Donsker-Varadhan (DV) representation for mutual information estimation and a Gradient Reversal Layer (GRL) for effective minimization. Adaptive Gradient Clipping is further employed to manage potentially unbounded mutual information gradients.

Experimental Evaluation

The efficacy of FDEN was validated across several diverse domains:

  • Faces: MS-Celeb-1M and CelebA (40 binary attributes).
  • Characters: Omniglot handwritten characters.
  • Natural Images: Mini-ImageNet.
  • Flowers: Oxford Flower dataset.
  • Synthetic Shapes: dSprites benchmark.

Results and Discussion

FDEN demonstrates significant capabilities in semantic image manipulation and few-shot learning. By replacing specific factors while maintaining others (e.g., keeping "identity" constant while changing "style"), the model enables precise attribute control across various image types.

Figure 2: Qualitative disentanglement results for faces, characters, and flowers, showing the separation of identity and style factors.

Quantitatively, FDEN achieved state-of-the-art performance on the dSprites benchmark with a Mutual Information Gap (MIG) of 0.5992, surpassing established models like $\beta$-VAE and FactorVAE. In few-shot classification on Omniglot, FDEN reached 95.4% accuracy (5-shot), significantly outperforming standard MLP baselines.

Conclusion and Limitations

The "plug-in" nature of FDEN allows for the enhancement of pre-trained models without retraining. However, its performance is bounded by the quality of the underlying model; features ignored during initial training cannot be recovered. Future work will explore more robust dependency measures, such as Wasserstein distance, to further refine factor independence.

Summary updated

Abstract

In this article, we focus on decomposing latent representations in generative adversarial networks or learned feature representations in deep autoencoders into semantically controllable factors in a semisupervised manner, without modifying the original trained models. Particularly, we propose factors' decomposer-entangler network (FDEN) that learns to decompose a latent representation into mutually independent factors. Given a latent representation, the proposed framework draws a set of interpretable factors, each aligned to independent factors of variations by minimizing their total correlation in an information-theoretic means. As a plug-in method, we have applied our proposed FDEN to the existing networks of adversarially learned inference and pioneer network and performed computer vision tasks of image-to-image translation in semantic ways, e.g., changing styles, while keeping the identity of a subject, and object classification in a few-shot learning scheme. We have also validated the effectiveness of the proposed method with various ablation studies in the qualitative, quantitative, and statistical examination.
BibTeX
@misc{yoon2019plug,
  author = {Jee Seok Yoon and Wonjun Ko and Heung-Il Suk},
  title = {{Plug-in Factorization for Latent Representation Disentanglement}},
  year = {2019},
  eprint = {1905.11088},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.11088},
}
Figure