Imagine trying to stretch a single, unbroken sheet of rubber to perfectly cover two separate islands. Because you are forbidden from cutting or tearing the rubber, you have no choice but to let it stretch across the open water between the islands.
This geometric dilemma is exactly what happens inside standard generative models like GANs, Normalizing Flows, and Diffusion Models. These models are designed as pushforward systems: they take a simple, completely connected latent space (like a single Gaussian or uniform distribution) and pass it through a deep neural network to generate complex data.
However, real-world data rarely sits on a single, unbroken manifold. As shown by Brown et al. in Verifying the Union of Manifolds Hypothesis for Image Data, image datasets are better understood as a union of disjoint, disconnected manifolds. Because neural networks are continuous functions, they are mathematically incapable of mapping a connected latent space to a disconnected target space without filling the empty gaps. This fundamental mismatch creates severe artifacts, training instabilities, and security vulnerabilities.
The Cost of Continuity: Artifacts and Instability
When a continuous generator is forced to bridge the gap between separate data clusters, it creates a variety of pathological behaviors across different modeling paradigms:
- Mode Connecting and Hallucinations: To maintain continuity, the generator must assign latent coordinates to the empty space between real data modes. As formalized by Khayatkhoei et al. in Disconnected Manifold Learning for Generative Adversarial Networks and the authors of Learning disconnected manifolds: a no GAN's land, this forces the model to output low-quality, unrealistic "bridge" samples. In score-based and diffusion models, Aithal et al. demonstrate in Understanding Hallucinations in Diffusion Models through Mode Interpolation that this smooth approximation behavior is the primary culprit behind structural hallucinations, creating impossible hybrid images where sharp discontinuities should exist.
- Density Leakage and Security Vulnerabilities: By filling the gaps between clusters with non-zero probability density, models fail to respect the true boundaries of the data manifold (Ross et al. in Neural Implicit Manifold Learning for Topology-Aware Density Estimation). Jang et al. reveal in On the Need for Topology-Aware Generative Models for Manifold-Based Defenses that these artificial high-density bridges create predictable pathways that adversarial attackers can exploit to bypass manifold-based defenses.
- Optimization Failure: High-dimensional, disconnected manifolds often suffer from a lack of support overlap. Mescheder et al. (Which Training Methods for GANs do actually Converge?) and Roth et al. (Stabilizing Training of Generative Adversarial Networks through Regularization) show that when the true and generated manifolds do not intersect, discriminators provide uninformative gradients, causing training to stagnate or collapse. Similarly, Hertrich et al. demonstrate in On the Relation between Rectified Flows and Optimal Transport that flow-based models frequently get trapped in non-optimal fixed points when trying to transport mass across disconnected clusters.
- Invalid Solutions in Inverse Problems: When generative models are used to solve ill-posed inverse problems, the continuity constraint can be disastrous. Chen et al. (Bayesian Inverse Problems with Conditional Sinkhorn Generative Adversarial Networks in Least Volume Latent Spaces) and Sun (Deep Probabilistic Imaging: Uncertainty Quantification and Multi-modal Solution Characterization for Computational Imaging) observe that when the true posterior distribution consists of isolated, valid solutions, continuous models interpolate across the gaps, generating physically impossible intermediate states.
Conversely, the problem can also occur in reverse: when algorithms designed strictly for discrete labels are forced to visualize continuous transitions, they can cause artificial "tears" or gaps in the manifold, as noted by the authors of Supervised dimensionality reduction technique accounting for soft classes.
To fix the continuous-to-disconnected mapping gap, researchers have developed three main strategies: space partitioning, latent space alignment, and post-hoc filtering.
Strategy 1: Space Partitioning and Piecewise Continuity
If a single continuous function cannot map to disconnected clusters, the most intuitive solution is to use multiple functions. By breaking the generative task into a mixture of sub-networks, models can construct a piecewise continuous mapping.
Single Generator (Continuous Bridge):
[ Connected Latent Space ] ---> ( Continuous Function ) ---> [ Cluster A ] === (Bridge Artifacts) ===> [ Cluster B ]
Multiple Generators (Piecewise Mapping):
/--> ( Generator 1 ) ---------> [ Cluster A ]
[ Connected Latent Space ] |
\--> ( Generator 2 ) ---------> [ Cluster B ]
Armandpour et al. implement this "divide and conquer" strategy in Partition-Guided GANs by partitioning complex data distributions into simpler, localized sub-regions and assigning a dedicated generator to each. This allows the framework to bypass the global continuity constraint entirely.
Similarly, Khayatkhoei et al. utilize a collection of generators alongside a learned prior that automatically determines how many sub-networks are required to cover the distinct clusters without creating interpolation artifacts.
Strategy 2: Structuring and Aligning the Latent Space
Instead of changing the architecture, another approach alters the input. If the latent space itself is structured to mirror the disconnected topology of the target data, the generator no longer needs to stretch or interpolate across empty space.
- Mixture Priors: Xiao et al. introduce this concept in BourGAN: Generative Networks with Metric Embeddings by replacing the standard unimodal Gaussian prior with a Gaussian mixture model constructed via metric embedding theory to match the data's true geometric clusters.
- Geometric Truncation: Issenhuth proves in Unveiling the Latent Space Geometry of Push-Forward Generative Models that the mathematically optimal way for a pushforward model to organize its latent space is through a Voronoi-like "simplicial cluster" geometry. Enforcing this structure via latent truncation prevents the generation of out-of-distribution samples.
- Optimal Transport Pre-mapping: An et al. develop a hybrid approach in AE-OT-GAN: Training GANs from Data Specific Latent Distribution. They use Optimal Transport to map a simple uniform prior to a data-specific, disconnected latent distribution before feeding it to the generator, giving the network a topologically accurate "warm start."
- Conditional Conditioning: As explored by Jimenez et al. in Generative Adversarial Network Performance in Low-Dimensional Settings, providing explicit conditional labels breaks the continuous bridge between modes by informing the network of distinct categorical boundaries during the mapping process.
Strategy 3: Post-Hoc Filtering and Latent Traversal
A final alternative accepts that the continuous model will inevitably build bridges during training, but focuses on detecting and avoiding those regions during inference.
The authors of Learning disconnected manifolds: a no GAN's land show that the "bridges" between modes correspond to regions where the generator must stretch drastically, which manifests as a massive spike in the network's Jacobian norm. By calculating this norm post-training, they can truncate and filter out low-quality transitional samples.
In a similar vein, Aithal et al. discover that diffusion models exhibit extreme prediction instability during the final sampling steps when they are trapped on an interpolation bridge, providing a natural metric to identify and discard hallucinations.
Finally, instead of filtering samples after generation, Coeurdoux et al. propose Normalizing flow sampling with Langevin dynamics in the latent space. They use a specialized Markov Chain Monte Carlo (MCMC) sampler directly within the latent space of a pre-trained model, navigating around the pathological bridge regions entirely without requiring any network retraining.
Key Foundational Work
Disconnected Manifold Learning for Generative Adversarial Networks (2018) — contributed 4 concept elements to this direction.
BourGAN: Generative Networks with Metric Embeddings (2018) — contributed 1 concept elements to this direction.
Which Training Methods for GANs do actually Converge? (2018) — contributed 1 concept elements to this direction.