TC-VAE: Uncovering Out-of-Distribution Data Generative Factors

Justin DauwelsarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces TC-VAE, a novel generative model that can discover "Out-of-Distribution" (OOD) generative factors – variations not explicitly present in the training data. This is a significant step towards achieving human-like generalization in AI, as it allows models to understand and represent concepts beyond what they've directly seen.

What Problem Does This Paper Address?

Disentanglement learning aims to identify the underlying factors that generate data (e.g., shape, color, position). While many models can learn these "in-distribution" factors, they struggle to uncover factors that are not represented or are imbalanced in the training dataset. This limitation hinders their ability to generalize and understand the world in a human-like way, as real-world datasets rarely have perfectly balanced and complete sets of generative factors. The paper tackles this by developing a method that can identify and represent these "missing" or under-represented generative factors.

Methods

The core of TC-VAE lies in its novel objective function, which is a lower bound on the joint total correlation (TC) between the learned latent representations and the input data. This bound is a convex combination of two key components:

  • Variational Information Bottleneck (VIB): This term encourages the latent representation to be informative about the data while also being compressed.
  • Conditional Information Bottleneck (CEB): This term, derived from multi-view representation learning, promotes a balanced distribution of information across all latent variables. This is crucial for handling imbalanced or missing generative factors.

By optimizing this combined lower bound, TC-VAE is encouraged to learn a more complete and balanced representation, enabling it to uncover OOD generative factors.

Datasets

The models were trained and evaluated on the following datasets:

  • Teapots: A synthetic dataset with 5 uniformly distributed generative factors (azimuth, elevation, and RGB color channels).
  • 3DShapes: A synthetic dataset with 6 uniformly distributed generative factors (shape, object color, object hue, floor color, wall color, and horizontal orientation).
  • Celeba: A real-world dataset of celebrity faces with diverse poses, expressions, and lighting conditions.
  • U3DShapes: A newly introduced unbalanced version of the 3DShapes dataset, where one generative factor (shape) is imbalanced to study the model's robustness.

Key Insights and Ideas

  • Uncovering the "Unseen": The most striking insight is TC-VAE's ability to discover generative factors that were not explicitly present or balanced in the training data. This is a significant leap towards more robust and generalizable AI.
  • The Power of Balanced Information: The CEB term in the objective function is key. By forcing latent variables to share information more equally, it prevents the model from ignoring less frequent or "missing" factors.
  • Real-World Data is Messy: The paper highlights that real-world datasets are unlikely to have perfectly balanced generative factors, a common assumption in many existing disentanglement methods. This imbalance can lead to degenerate solutions where models ignore under-represented factors.
  • t-SNE as a Visualization Tool: The authors propose using t-SNE plots of latent traversals to visualize learned generative factors, including OOD ones, providing an intuitive way to assess disentanglement.

Key Results

  • OOD Factor Discovery: TC-VAE successfully uncovered OOD generative factors on the 3DShapes and Teapots datasets, as visualized by latent traversals. For instance, it identified "vertical perspective" and "object position" as OOD factors on the 3DShapes dataset.
  • Quantitative Performance: TC-VAE consistently outperformed baseline models (B-VAE and FactorVAE) on various downstream metrics, including DC score and informativeness, across multiple datasets. For example, on the 3DShapes dataset, TC-VAE(α=0.25) achieved a DC score of 0.311, compared to FactorVAE's 0.219.
  • Robustness to Imbalance: While FactorVAE showed some promise on unbalanced datasets, TC-VAE generally maintained better informativeness metrics, though DCI scores became less reliable on severely unbalanced data, suggesting a need for further metric investigation.
  • Qualitative Improvements: Generated OOD samples from TC-VAE showed better reconstruction quality compared to baselines, even when all models could generate samples with the OOD attributes.

Limitations and Future Work

  • Disentanglement vs. Balance Trade-off: The CEB term, while crucial for balance and OOD discovery, can sometimes decrease the overall disentanglement of the model. This is because less informative factors might be mixed to achieve the desired balance.
  • Metric Inconsistency: The authors observed that downstream metrics like DCI can become inconsistent when dealing with unbalanced datasets, indicating a need for further research into robust evaluation methods.
  • Future Directions: The authors suggest further investigating the impact of generative factor imbalance on disentanglement learning and downstream metrics. They also propose applying TC-VAE to real-world structured data (e.g., protein structures) to potentially uncover unknown generative factors and generate novel structures.

Summary updated

Abstract

Uncovering data generative factors is the ultimate goal of disentanglement learning. Although many works proposed disentangling generative models able to uncover the underlying generative factors of a dataset, so far no one was able to uncover OOD generative factors (i.e., factors of variations that are not explicitly shown on the dataset). Moreover, the datasets used to validate these models are synthetically generated using a balanced mixture of some predefined generative factors, implicitly assuming that generative factors are uniformly distributed across the datasets. However, real datasets do not present this property. In this work we analyse the effect of using datasets with unbalanced generative factors, providing qualitative and quantitative results for widely used generative models. Moreover, we propose TC-VAE, a generative model optimized using a lower bound of the joint total correlation between the learned latent representations and the input data. We show that the proposed model is able to uncover OOD generative factors on different datasets and outperforms on average the related baselines in terms of downstream disentanglement metrics.
BibTeX
@misc{dauwels2023tc,
  author = {Justin Dauwels},
  title = {{TC-VAE: Uncovering Out-of-Distribution Data Generative Factors}},
  year = {2023},
  eprint = {2304.04103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.04103},
}
Figure