A Plug-in Method for Representation Factorization in Connectionist Models

Jee Seok Yoon, Myung-Cheol Roh, Heung-Il Suk

Abstract

The Factors Decomposer-Entangler Network (FDEN) is a novel "plug-in" framework designed to disentangle the latent representations of pre-trained deep models into independent, semantically meaningful factors. By minimizing total correlation through information-theoretic means, FDEN allows researchers to perform complex tasks like style transfer and few-shot learning without modifying or retraining the original, high-performing model. This approach bridges the gap between the high performance of "black-box" models and the need for interpretable, controllable latent spaces.

Introduction

Deep learning models, particularly Generative Adversarial Networks (GANs) and Autoencoders, excel at learning compressed latent representations of data. However, these representations are typically entangled: a single numerical value in the latent vector might influence multiple attributes simultaneously, such as the identity of a face and the lighting of the scene.

Traditionally, achieving "disentangled" representations—where one factor controls only one attribute—requires designing and training entirely new architectures from scratch (e.g., $\beta$-VAE). This is inefficient when state-of-the-art models already exist. FDEN addresses this by acting as a modular wrapper around a fixed, pre-trained model, factorizing its latent space into controllable parts without altering the original model's weights.

The FDEN Framework

The FDEN architecture operates as a post-processing module consisting of three main components:

  1. Decomposer ($\mathcal{D}$): Maps the original latent vector $z$ into a set of factors $f_0, f_1, ..., f_N$.
  2. Factorizer ($\mathcal{F}$): The core component that ensures independence. It utilizes a Statisticians Network to estimate and minimize the Total Correlation (a multi-variable version of mutual information) between factors. Small Alignment Networks (classifiers) link specific factors to human-labeled attributes.
  3. Entangler ($\mathcal{E}$): Recombines these factors back into a reconstructed version of the original latent vector ($\tilde{z}$).
The detailed architecture of FDEN, showing the Decomposer (a), the Factorizer (b) with its Gradient Reversal Layer (GRL), and the Entangler (c).

To ensure stable training, the authors employ a Gradient Reversal Layer (GRL) and Adaptive Gradient Clipping, allowing the model to effectively "push" the factors apart until they achieve statistical independence.

Experimental Evaluation

The researchers evaluated FDEN across a diverse range of domains to demonstrate its versatility:

  • Faces: MS-Celeb-1M and CelebA datasets for identity and attribute control.
  • Characters: The Omniglot dataset for handwritten character analysis.
  • Natural Images: Mini-ImageNet and Oxford Flower datasets.
  • Synthetic Benchmarks: dSprites for quantitative disentanglement scoring.

Key Results

FDEN demonstrates significant utility in several downstream tasks:

  • Style Transfer: On the MS-Celeb-1M and Oxford Flower datasets, FDEN successfully separated "identity" from "style." By interpolating the style factor while holding the identity factor constant, the model generated smooth transitions in lighting and background without changing the subject's core identity.
  • Few-Shot Learning: FDEN improved classification performance on unseen classes. By using the factorized identity vector, an MLP classifier achieved 88.3% accuracy on 1-shot Omniglot, compared to 80.3% using the raw, entangled latent vector.
  • Disentanglement Performance: On the dSprites dataset, FDEN achieved disentanglement scores (BVM, FVM, MIG) competitive with state-of-the-art native architectures like FactorVAE, despite being a plug-in method.
Visual results of style transfer across faces, characters, and flowers, demonstrating the separation of identity (f_ide) and style (f_sty).

Discussion and Limitations

While FDEN offers a powerful way to add interpretability to existing models, it is inherently upper-bounded by the base model's performance; it cannot recover information the original model failed to capture. Additionally, forcing independence between naturally correlated human labels (e.g., age and baldness) can lead to unintentional artifacts during factor manipulation. Future work may explore more granular dependency measures, such as the Wasserstein dependency measure, to further refine latent space factorization.

Summary updated

Abstract

In this article, we focus on decomposing latent representations in generative adversarial networks or learned feature representations in deep autoencoders into semantically controllable factors in a semisupervised manner, without modifying the original trained models. Particularly, we propose factors’ decomposer-entangler network (FDEN) that learns to decompose a latent representation into mutually independent factors. Given a latent representation, the proposed framework draws a set of interpretable factors, each aligned to independent factors of variations by minimizing their total correlation in an information-theoretic means. As a plug-in method, we have applied our proposed FDEN to the existing networks of adversarially learned inference and pioneer network and performed computer vision tasks of image-to-image translation in semantic ways, e.g., changing styles, while keeping the identity of a subject, and object classification in a few-shot learning scheme. We have also validated the effectiveness of the proposed method with various ablation studies in the qualitative, quantitative, and statistical examination.
BibTeX
@article{yoon2022plug,
  author = {Jee Seok Yoon and Myung-Cheol Roh and Heung-Il Suk},
  title = {{A Plug-in Method for Representation Factorization in Connectionist Models}},
  journal = {IEEE TNNLS},
  year = {2022},
  doi = {10.1109/tnnls.2021.3054480},
}
Figure