Learning Bijective Feature Maps for Linear ICA

Alexander Camuto, Matthew Willetts, Brooks Paige, Chris C. Holmes, Stephen J. Roberts

Abstract

Independent Component Analysis (ICA) is a fundamental technique for source separation, yet its linear formulation struggles with complex, high-dimensional data like images. We present Bijecta, a framework that leverages bijective flows to map data into a feature space where linear ICA can effectively identify independent latent factors. By incorporating Johnson-Lindenstrauss projections to stabilize training, Bijecta achieves superior disentanglement compared to standard Variational Autoencoders (VAEs) and normalizing flows, as demonstrated on benchmarks like CelebA and dSprites.

Introduction

Independent Component Analysis (ICA) is a classic tool in signal processing, famously used for the "cocktail party problem"—separating individual voices from a noisy room. However, when it comes to high-dimensional, non-linearly mixed data like images, traditional linear ICA often fails. The goal of representation learning is to transform raw data into "latent factors" that correspond to interpretable physical attributes, such as rotation or lighting.

Existing approaches face significant hurdles. Linear ICA assumes simple linear mixtures, which is insufficient for image data. Conversely, Deep Generative Models like Variational Autoencoders (VAEs) can handle complexity but often fail to "disentangle" factors, resulting in correlated latent variables. Normalizing Flows offer exact likelihood calculation but are typically "square," requiring the latent space to match the input dimensionality—an inefficient constraint for image compression.

Methodology: The Bijecta Framework

Bijecta addresses these challenges by combining the expressive power of deep generative models with the interpretability of ICA. The architecture operates in two primary stages:

  1. Bijective Flow: An invertible neural network (utilizing Rational Quadratic Splines) transforms raw data $x$ into a feature space $z$ of the same dimension. This flow "pre-processes" the data to resemble a linear mixture.
  2. Linear ICA Layer: This layer maps the high-dimensional $z$ down to a lower-dimensional space of independent sources $s$.
Graphical model of Bijecta showing the transformation from data $x$ to sources $s$ via latent space $z$.

To ensure stability during training, the authors introduce Johnson-Lindenstrauss (JL) Projections. Training a powerful flow alongside a linear ICA model is notoriously unstable, as the flow can "overpower" the ICA layer. Instead of using unstable data-driven whitening, Bijecta employs fixed, random JL matrices that are approximately orthonormal. This anchors the model, forcing the flow to organize data into a structure that the fixed ICA layer can easily unmix. Furthermore, the model utilizes non-Gaussian priors (e.g., Laplace distributions) to force the discovery of axis-aligned, independent factors.

Experimental Results

The researchers evaluated Bijecta on standard benchmarks including dSprites, CelebA, CIFAR-10, and MNIST.

Independence and Disentanglement

On the CelebA dataset, Bijecta achieved a Total Correlation (TC)—a metric for shared information between latent variables—of 13.1. This significantly outperformed standard Laplace-VAEs (106.7) and $\beta$-TCVAEs (55.7), indicating that Bijecta’s factors are far more independent.

Comparison of latent factor discovery between VAEs and Bijecta on a toy heart dataset. Bijecta (c) successfully unmixes independent factors like rotation and position, whereas VAEs (b) show entangled, blurry results.

Latent Traversals

Visual inspections of latent space "walks" revealed that Bijecta captures consistent, specific attributes. In face datasets, traversing a single dimension might change only hair thickness or gender, whereas VAEs often tangle these attributes together. Additionally, despite the bijective nature of the flow, the model naturally concentrates variance into a few dimensions, providing effective dimensionality reduction.

Discussion and Limitations

While Bijecta demonstrates strong performance in factor discovery, it faces a trade-off between perfect reconstruction and perfect independence. Because it performs non-square ICA for dimensionality reduction, it cannot achieve the lossless reconstruction of standard square flows. Furthermore, while the model moves toward the "holy grail" of non-linear identifiability—the guarantee of finding true sources—a full mathematical proof in non-linear settings remains an open challenge. Future work may explore deeper flows and varied non-Gaussian priors to capture even more subtle independent factors in diverse data modalities.

Summary updated

Abstract

Separating high-dimensional data like images into independent latent factors, i.e independent component analysis (ICA), remains an open research problem. As we show, existing probabilistic deep generative models (DGMs), which are tailor-made for image data, underperform on non-linear ICA tasks. To address this, we propose a DGM which combines bijective feature maps with a linear ICA model to learn interpretable latent structures for high-dimensional data. Given the complexities of jointly training such a hybrid model, we introduce novel theory that constrains linear ICA to lie close to the manifold of orthogonal rectangular matrices, the Stiefel manifold. By doing so we create models that converge quickly, are easy to train, and achieve better unsupervised latent factor discovery than flow-based models, linear ICA, and Variational Autoencoders on images.
BibTeX
@inproceedings{camuto2021learning,
  author = {Alexander Camuto and Matthew Willetts and Brooks Paige and Chris C. Holmes and Stephen J. Roberts},
  title = {{Learning Bijective Feature Maps for Linear ICA}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=a5v2_9vBFN},
}
Figure