Abstract
Separating high-dimensional data like images into independent latent factors, i.e independent component analysis (ICA), remains an open research problem. As we show, existing probabilistic deep generative models (DGMs), which are tailor-made for image data, underperform on non-linear ICA tasks. To address this, we propose a DGM which combines bijective feature maps with a linear ICA model to learn interpretable latent structures for high-dimensional data. Given the complexities of jointly training such a hybrid model, we introduce novel theory that constrains linear ICA to lie close to the manifold of orthogonal rectangular matrices, the Stiefel manifold. By doing so we create models that converge quickly, are easy to train, and achieve better unsupervised latent factor discovery than flow-based models, linear ICA, and Variational Autoencoders on images.
Introduction
Independent Component Analysis (ICA) is often described as the "cocktail party problem": the goal is to recover original, independent source signals from a complex mixture. While Linear ICA is effective for simple audio signals, it struggles with high-dimensional data like images, where the relationship between latent factors (e.g., object rotation, lighting) and raw pixels is highly non-linear.
Modern Deep Generative Models (DGMs), such as Variational Autoencoders (VAEs), excel at modeling image distributions but frequently fail to "disentangle" latent factors into truly independent sources. Furthermore, most flow-based models are "square," requiring the number of latent variables to match the input dimensionality, which limits their utility for dimensionality reduction. This paper introduces Bijecta, a hybrid model designed to bridge the gap between non-linear flexibility and linear interpretability.
Methodology
Bijecta decomposes the generative process into two jointly trained components:
- Bijective Flow: An 8-layer flow utilizing Rational Quadratic Splines (RQS) maps high-dimensional input $X$ into a feature space $Z$ of identical dimension. This flow learns a representation where underlying sources are linearly mixed.
- Linear ICA Model: This component maps the feature space $Z$ down to a lower-dimensional source space $S$, performing non-square ICA.
Jointly training these components is inherently unstable. To address this, the authors constrain the linear unmixing matrix to the Stiefel manifold—the space of orthonormal rectangular matrices. Rather than employing computationally expensive Singular Value Decomposition (SVD) at each step, Bijecta utilizes a fixed, random Johnson-Lindenstrauss (JL) projection. This "sketching" technique ensures proper whitening and decorrelation, allowing the flow to focus on learning the non-linear mapping without the overhead of traditional whitening methods.
Experiments and Results
The model was evaluated on benchmarks including dSprites, CelebA, Fashion-MNIST, and CIFAR-10.
Latent Factor Discovery
On the dSprites dataset, Bijecta successfully recovered axis-aligned physical transformations. While VAEs often produce "kinks" or non-linear paths in latent space, Bijecta's sources correspond to linear movements (e.g., strictly horizontal or vertical translation).
Quantitative Performance
In CelebA experiments, Bijecta demonstrated superior disentanglement. It achieved a Total Correlation (TC)—a metric for latent dependence where lower is better—of 13.1. This significantly outperformed a standard Laplace-VAE (106.7) and the state-of-the-art $\beta$-TCVAE (55.7).
Dimensionality Reduction
Unlike standard flows that distribute information across all dimensions, Bijecta's joint objective forces variance into a small number of latent dimensions. On Fashion-MNIST and CIFAR-10, the model concentrated nearly all explained variance into the first 64 dimensions, whereas standard RQS flows required hundreds of dimensions for comparable information capture.
Discussion
Bijecta demonstrates that VAEs struggle to match aggregate posteriors to ICA-appropriate priors (like the heavy-tailed Laplace distribution), leading to entangled latent spaces. By using a flow to "preprocess" data for a linear ICA model, Bijecta achieves a more interpretable description of the data manifold.
A key takeaway is the importance of geometric stability; staying near the Stiefel manifold is critical for convergence. While the model is non-square and thus not perfectly invertible—requiring $L_1$ regularization for high-fidelity reconstructions—it offers a powerful framework for unsupervised source separation in high-dimensional domains. Future work may explore incorporating auxiliary variables to provide formal identifiability guarantees, which non-linear ICA generally lacks without additional constraints.
Summary updated