Many generative models assume that images lie on one smooth, connected surface (a "manifold") inside pixel space. The evidence says otherwise. Natural image datasets look more like a collection of separate surfaces, and these surfaces differ in dimension: some ImageNet classes have an estimated intrinsic dimension near 15 and others near 35 (Verifying the Union of Manifolds Hypothesis for Image Data). Injective flows are the family of models built to place probability exactly on a low-dimensional manifold. An injective flow is an invertible-on-its-image network that maps a $d$-dimensional latent space into pixel space and still gives an exact likelihood. Every existing injective flow uses a single latent space of one fixed dimension. It therefore has to bridge the gaps between components, and it has to give all of them the same dimension.
We propose to fix this in the first, geometry-only stage of the standard two-stage recipe (first learn where the data is, then learn how it is distributed there). During that stage, the autoencoder's latent codes are pushed toward a union of linear subspaces. The push comes from a block-diagonal self-expression penalty taken from subspace clustering. This penalty asks each code to be rebuilt from codes in its own group only, and it penalizes the affinity graph unless it splits into $k$ disconnected pieces. Each subspace that emerges gives three things: a cluster found without labels, its own dimension $d_k$ read off from its rank, and a small injective map from $\mathbb{R}^{d_k}$ to pixel space. That map has a cheap exact volume term, so a separate low-dimensional flow can model the density on each component.
Core Claim
The two-stage literature contains an unresolved conflict. Decoupling geometry from density provably avoids degenerate likelihood spikes (Diagnosing and Fixing Manifold Overfitting in Deep Generative Models; Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows). Yet Conformal Embedding Flows found that strictly sequential training on CelebA gave FID 171, against 118 for joint training. Their explanation is that an autoencoder trained only on reconstruction places the data in a latent layout that is hard for the flow to model.
Our claim is that this hostility is largely a structure problem. A reconstruction-only latent mixes components of different dimensions in one shared space. A geometric stage that is explicitly regularized to produce a union of low-dimensional subspaces should give latents that are easy to model one component at a time. If so, the two-stage method keeps its stability guarantee and loses most of its sample-quality penalty. It also gains two things no current injective flow has: unsupervised discovery of components and a dimension per component.
The idea differs from its nearest neighbours in specific ways. Deep Subspace Clustering with Block Diagonal Constraint uses the same kind of penalty inside an autoencoder, but only for clustering. It has no density, no sampling, and no injectivity. The Disconnected generative models of Brown et al. (The Union of Manifolds Hypothesis) do model components separately, but they need class labels to split the data, and they call unsupervised splitting an open problem. Here the penalty is a tool for shaping geometry so that the later density stage works, and clustering comes out as a by-product.
Research Program
Stage 1 (geometry). An encoder $E$ and decoder $G$ are trained on reconstruction error plus two self-expression terms on the latent codes $z_i = E(x_i)$. The full penalty of Subspace Clustering by Block Diagonal Representation needs an $N \times N$ matrix and costs $O(N^3)$, so we use a dictionary version instead, in the spirit of Large-Scale Subspace Clustering via k-Factorization. There are $M$ learned anchor codes $A = [a_1,\dots,a_M]$. Each minibatch code is written as $z_i \approx A c_i$. The penalty is applied to the $M \times M$ anchor affinity $B$, which is built from the anchors' own self-expression. The objective is
$$ \mathcal{L}_1 = \mathbb{E}\|x - G(E(x))\|^2 + \alpha\, \mathbb{E}\|E(x) - A c(x)\|^2 + \beta \sum_{j=1}^{k} \sigma_j(L_B), $$
where $L_B$ is the graph Laplacian of $B$ and $\sigma_j$ are its $k$ smallest eigenvalues. The last term is zero exactly when the anchor graph has at least $k$ connected components. The eigen-decomposition then costs $O(M^3)$ with $M$ in the low thousands, so it does not depend on dataset size.
When training ends, each connected block of anchors spans a subspace $S_k$. We choose its dimension $d_k$ from the singular-value decay of the codes assigned to it. We fit an orthonormal basis $U_k \in \mathbb{R}^{d\times d_k}$. Each point is assigned to the component with the lowest reconstruction error after projection onto $S_k$.
Stage 2 (density). The encoder, decoder and bases are frozen. For each component we train a standard $d_k$-dimensional flow on the coordinates $u = U_k^\top E(x)$. The component's pushforward density on $\mathcal{M}_k = G(S_k)$ uses the generalized change of variables:
$$ p_k(x) = q_k(u)\,\det\!\big(J_k(u)^\top J_k(u)\big)^{-1/2}, \qquad J_k(u) = \tfrac{\partial}{\partial u} G(U_k u), $$
where $q_k$ is the latent flow density. Because $d_k$ is roughly 10 to 40, the $d_k \times d_k$ Gram matrix can be computed exactly at evaluation time with $d_k$ Jacobian-vector products. No conformal restriction is needed, and neither is the surrogate estimator of Lifting Architectural Constraints of Injective Flows. We will check injectivity of $G$ restricted to each $S_k$ numerically by monitoring the smallest singular value of $J_k$. If it collapses, we fall back to the zero-padding decoder of Flows for simultaneous manifold learning and density estimation.
Sampling first draws a component with weight $\pi_k$ equal to its empirical share, then samples from that component. One honest caveat follows. Densities on manifolds of different dimension are defined with respect to different volume measures. A $d_k = 15$ likelihood is not comparable to a $d_k = 30$ likelihood. We therefore report likelihoods within each component, and we never sum mixture log-likelihoods across dimensions. Comparisons between models rely on sample metrics and on synthetic data where the true density is known.
The analysis question running through the project is whether the penalty makes the latent easier to model. We test this directly. The measure is the held-out negative log-likelihood of a flow with fixed capacity, trained on the Stage-1 latents, with and without the penalty.
Baselines
The naive reference point is the plain two-stage model. It has a reconstruction-only autoencoder followed by a single latent flow, which is the Generalized Autoencoder recipe of Diagnosing and Fixing Manifold Overfitting in Deep Generative Models. It uses the same encoder, decoder and total flow parameters as our method. Every gain we claim must first appear against this model, because it isolates what the penalty adds.
The strongest single-manifold injective flow is Free-Form Injective Flows. These are trained jointly and handle curvature by evaluating the volume change off the manifold. They report CelebA FID 37.4 (Lifting Architectural Constraints of Injective Flows). Trumpets report 34.3 in their own paper (Trumpets), while the FIF paper quotes Trumpets at 47.7. We will rerun both with released settings rather than rely on either number. We will also include joint and sequential Conformal Embedding Flows (FID 118 and 171 on CelebA, Conformal Embedding Flows), because the sequential-versus-joint gap is the effect we aim to close.
For the union-of-manifolds setting, the strongest competitor is the label-partitioned Disconnected WAE. It improved CIFAR-10 FID from 146.7 to 133.3 (Verifying the Union of Manifolds Hypothesis for Image Data). It acts as a supervised oracle for partitioning. Beating it means matching or improving 133.3 under the same parameter and FLOP budget without labels. Matching it alone would already show that labels are unnecessary.
For clustering, the comparison is DSC-BDC and linear BDR on Extended Yale B, ORL and COIL20. The file gives only relative gains for DSC-BDC: up to 0.74% NMI and 0.73% ARI over DSC-Net on Extended Yale B, with no absolute accuracy numbers found on Lacuna. We will rerun DSC-BDC ourselves.
The key ablation is the penalty applied after the fact instead of during Stage 1. Here a plain autoencoder is trained, then BDR or k-means is run on frozen latents, then per-cluster flows are fitted. Its components get dimensions from the same rank estimate.
All models get the same data splits, an equal hyperparameter-search budget (a fixed number of trials per method), and matched GPU-hours.
Why not just…?
Why not just cluster the latents after training a plain autoencoder? This is the real threat. If post-hoc clustering plus per-cluster flows matches our method, the contribution shrinks to an engineering recipe, and we would say so. Our hypothesis is that a reconstruction-only latent is not a union of low-rank pieces. Clusters found afterwards would then have inflated, poorly separated ranks, and the per-cluster flows would face the same hostile geometry that Ross et al. observed. The post-hoc ablation measures this directly, through per-component effective rank, held-out latent NLL at fixed flow capacity, and FID.
Why not just use one injective flow with a mixture-of-Gaussians base distribution? A mixture base can separate modes, but every mode still shares the same dimension $d$, and the decoder still has to stretch one connected latent across disconnected components. Brown et al. show that continuous models bridge empty space between components. Our synthetic benchmark, a union of a 2-D surface and a 6-D surface with gaps between them, will measure the mass this baseline places off the true support.
Why not just partition by class labels, as in Disconnected generative models? Labels are unavailable in most density-estimation settings, and Brown et al. themselves call them an imperfect proxy for manifold structure. We include the label partition as an oracle. If our unsupervised components improve on it, that suggests classes are not the right units. If they fall short, the size of the gap is the cost of having no labels.
Why not just train jointly, as CEF and FIF do? Joint training gives up the provable protection against likelihood spikes, and it still assumes one manifold. If joint FIF beats us on CelebA, a dataset where union structure is weaker, we would treat that as a limit on where our method applies, not as a refutation. The decisive regime is datasets with many components and varied dimension.
Proposed Experiments
Synthetic union of manifolds. The first experiment uses several curved surfaces of different dimensions (1 to 8), embedded in 64 ambient dimensions with known densities, including pairs that are close in angle. Metrics are per-component KL divergence to the true density, estimated against the known density; the fraction of samples farther than $\epsilon$ from every true component; and recovery of $d_k$.
Clustering as a sanity check. On Hopkins 155, Extended Yale B, ORL and COIL20, the Stage-1 penalty should match DSC-BDC on accuracy, NMI and ARI. On Hopkins, linear BDR reports 0.93% mean error (Subspace Clustering by Block Diagonal Representation). This check shows the dictionary approximation does not discard the clustering ability. Generation quality is not the goal here.
Main generative test. On MNIST, Fashion-MNIST and CIFAR-10 we compare FID, per-component likelihood, and the agreement between estimated $d_k$ and the Levina–Bickel estimates of Brown et al. against the D-WAE oracle and the single-manifold flows.
The decisive check. We repeat the CEF sequential-versus-joint comparison on CelebA with our penalty added to the sequential arm. If the penalized two-stage model recovers most of the 171-to-118 gap, the core claim holds. The supporting evidence would be lower held-out latent NLL for the fixed-capacity flow and a FID clearly better than the plain two-stage baseline. If the post-hoc clustering ablation does as well, the claim that shaping geometry during Stage 1 matters is falsified, and the paper becomes a negative result about where block-diagonal structure helps.
References
- Verifying the Union of Manifolds Hypothesis for Image Data
- Diagnosing and Fixing Manifold Overfitting in Deep Generative Models
- Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows
- Conformal Embedding Flows: Tractable Density Estimation on Learned Manifolds
- Deep Subspace Clustering with Block Diagonal Constraint
- The Union of Manifolds Hypothesis
- Subspace Clustering by Block Diagonal Representation
- Large-Scale Subspace Clustering via k-Factorization
- Lifting Architectural Constraints of Injective Flows
- Flows for simultaneous manifold learning and density estimation
- Trumpets: Injective Flows for Inference and Inverse Problems