Abstract
Independent Component Analysis (ICA) is a fundamental technique for source separation, yet its linear formulation struggles with complex, high-dimensional data like images. We present Bijecta, a framework that leverages bijective flows to map data into a feature space where linear ICA can effectively identify independent latent factors. By incorporating Johnson-Lindenstrauss projections to stabilize training, Bijecta achieves superior disentanglement compared to standard Variational Autoencoders (VAEs) and normalizing flows, as demonstrated on benchmarks like CelebA and dSprites.
Introduction
Independent Component Analysis (ICA) is a classic tool in signal processing, famously used for the "cocktail party problem"—separating individual voices from a noisy room. However, when it comes to high-dimensional, non-linearly mixed data like images, traditional linear ICA often fails. The goal of representation learning is to transform raw data into "latent factors" that correspond to interpretable physical attributes, such as rotation or lighting.
Existing approaches face significant hurdles. Linear ICA assumes simple linear mixtures, which is insufficient for image data. Conversely, Deep Generative Models like Variational Autoencoders (VAEs) can handle complexity but often fail to "disentangle" factors, resulting in correlated latent variables. Normalizing Flows offer exact likelihood calculation but are typically "square," requiring the latent space to match the input dimensionality—an inefficient constraint for image compression.
Methodology: The Bijecta Framework
Bijecta addresses these challenges by combining the expressive power of deep generative models with the interpretability of ICA. The architecture operates in two primary stages:
- Bijective Flow: An invertible neural network (utilizing Rational Quadratic Splines) transforms raw data $x$ into a feature space $z$ of the same dimension. This flow "pre-processes" the data to resemble a linear mixture.
- Linear ICA Layer: This layer maps the high-dimensional $z$ down to a lower-dimensional space of independent sources $s$.
To ensure stability during training, the authors introduce Johnson-Lindenstrauss (JL) Projections. Training a powerful flow alongside a linear ICA model is notoriously unstable, as the flow can "overpower" the ICA layer. Instead of using unstable data-driven whitening, Bijecta employs fixed, random JL matrices that are approximately orthonormal. This anchors the model, forcing the flow to organize data into a structure that the fixed ICA layer can easily unmix. Furthermore, the model utilizes non-Gaussian priors (e.g., Laplace distributions) to force the discovery of axis-aligned, independent factors.
Experimental Results
The researchers evaluated Bijecta on standard benchmarks including dSprites, CelebA, CIFAR-10, and MNIST.
Independence and Disentanglement
On the CelebA dataset, Bijecta achieved a Total Correlation (TC)—a metric for shared information between latent variables—of 13.1. This significantly outperformed standard Laplace-VAEs (106.7) and $\beta$-TCVAEs (55.7), indicating that Bijecta’s factors are far more independent.
Latent Traversals
Visual inspections of latent space "walks" revealed that Bijecta captures consistent, specific attributes. In face datasets, traversing a single dimension might change only hair thickness or gender, whereas VAEs often tangle these attributes together. Additionally, despite the bijective nature of the flow, the model naturally concentrates variance into a few dimensions, providing effective dimensionality reduction.
Discussion and Limitations
While Bijecta demonstrates strong performance in factor discovery, it faces a trade-off between perfect reconstruction and perfect independence. Because it performs non-square ICA for dimensionality reduction, it cannot achieve the lossless reconstruction of standard square flows. Furthermore, while the model moves toward the "holy grail" of non-linear identifiability—the guarantee of finding true sources—a full mathematical proof in non-linear settings remains an open challenge. Future work may explore deeper flows and varied non-Gaussian priors to capture even more subtle independent factors in diverse data modalities.
Summary updated