Abstract
The Factors Decomposer-Entangler Network (FDEN) is a novel "plug-in" framework designed to disentangle the latent representations of pre-trained deep models into independent, semantically meaningful factors. By minimizing total correlation through information-theoretic means, FDEN allows researchers to perform complex tasks like style transfer and few-shot learning without modifying or retraining the original, high-performing model. This approach bridges the gap between the high performance of "black-box" models and the need for interpretable, controllable latent spaces.
Introduction
Deep learning models, particularly Generative Adversarial Networks (GANs) and Autoencoders, excel at learning compressed latent representations of data. However, these representations are typically entangled: a single numerical value in the latent vector might influence multiple attributes simultaneously, such as the identity of a face and the lighting of the scene.
Traditionally, achieving "disentangled" representations—where one factor controls only one attribute—requires designing and training entirely new architectures from scratch (e.g., $\beta$-VAE). This is inefficient when state-of-the-art models already exist. FDEN addresses this by acting as a modular wrapper around a fixed, pre-trained model, factorizing its latent space into controllable parts without altering the original model's weights.
The FDEN Framework
The FDEN architecture operates as a post-processing module consisting of three main components:
- Decomposer ($\mathcal{D}$): Maps the original latent vector $z$ into a set of factors $f_0, f_1, ..., f_N$.
- Factorizer ($\mathcal{F}$): The core component that ensures independence. It utilizes a Statisticians Network to estimate and minimize the Total Correlation (a multi-variable version of mutual information) between factors. Small Alignment Networks (classifiers) link specific factors to human-labeled attributes.
- Entangler ($\mathcal{E}$): Recombines these factors back into a reconstructed version of the original latent vector ($\tilde{z}$).
To ensure stable training, the authors employ a Gradient Reversal Layer (GRL) and Adaptive Gradient Clipping, allowing the model to effectively "push" the factors apart until they achieve statistical independence.
Experimental Evaluation
The researchers evaluated FDEN across a diverse range of domains to demonstrate its versatility:
- Faces: MS-Celeb-1M and CelebA datasets for identity and attribute control.
- Characters: The Omniglot dataset for handwritten character analysis.
- Natural Images: Mini-ImageNet and Oxford Flower datasets.
- Synthetic Benchmarks: dSprites for quantitative disentanglement scoring.
Key Results
FDEN demonstrates significant utility in several downstream tasks:
- Style Transfer: On the MS-Celeb-1M and Oxford Flower datasets, FDEN successfully separated "identity" from "style." By interpolating the style factor while holding the identity factor constant, the model generated smooth transitions in lighting and background without changing the subject's core identity.
- Few-Shot Learning: FDEN improved classification performance on unseen classes. By using the factorized identity vector, an MLP classifier achieved 88.3% accuracy on 1-shot Omniglot, compared to 80.3% using the raw, entangled latent vector.
- Disentanglement Performance: On the dSprites dataset, FDEN achieved disentanglement scores (BVM, FVM, MIG) competitive with state-of-the-art native architectures like FactorVAE, despite being a plug-in method.
Discussion and Limitations
While FDEN offers a powerful way to add interpretability to existing models, it is inherently upper-bounded by the base model's performance; it cannot recover information the original model failed to capture. Additionally, forcing independence between naturally correlated human labels (e.g., age and baldness) can lead to unintentional artifacts during factor manipulation. Future work may explore more granular dependency measures, such as the Wasserstein dependency measure, to further refine latent space factorization.
Summary updated