Geometric Enclosing Networks

Trung Le, Hung Vu, Tu Dinh Nguyen, Dinh Q. Phung

Abstract

Training models to generate data has increasingly attracted research attention and become important in modern world applications. We propose in this paper a new geometry-based optimization approach to address this problem. Orthogonal to current state-of-the-art density-based approaches, most notably VAE and GAN, we present a fresh new idea that borrows the principle of minimal enclosing ball to train a generator $G(z)$ in such a way that both training and generated data, after being mapped to the feature space, are enclosed in the same sphere. We develop theory to guarantee that the mapping is bijective so that its inverse from feature space to data space results in expressive nonlinear contours to describe the data manifold, hence ensuring data generated are also lying on the data manifold learned from training data. Our model enjoys a nice geometric interpretation, hence termed Geometric Enclosing Networks (GEN), and possesses some key advantages over its rivals, namely simple and easy-to-control optimization formulation, avoidance of mode collapsing and efficiently learn data manifold representation in a completely unsupervised manner. We conducted extensive experiments on synthesis and real-world datasets to illustrate the behaviors, strength and weakness of our proposed GEN, in particular its ability to handle multi-modal data and quality of generated data.

Introduction

Most modern generative models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), are rooted in density estimation. They attempt to approximate the underlying probability distribution of the data. However, this approach introduces significant challenges: GANs are notoriously difficult to train, often suffering from convergence issues and "mode collapse"—where the generator produces only a limited subset of data types (e.g., generating only one specific digit when trained on MNIST).

Geometric Enclosing Networks (GEN) offer a fresh alternative by shifting focus from density estimation to geometry. Instead of modeling probability densities, GEN focuses on the geometric manifold—the "shape" that the data occupies in space. By training a generator to fit data within a "minimal enclosing ball" in a high-dimensional feature space, GEN effectively eliminates mode collapse and provides a more stable, easier-to-control optimization process.

Methods

The core principle of GEN is derived from classical kernel methods: the Minimal Enclosing Ball (MEB). The framework consists of several key components:

  1. Feature Mapping ($\Phi$): In kernel methods, data is mapped into a high-dimensional feature space where complex patterns become simpler geometric shapes. To enable backpropagation in a deep learning context, GEN utilizes Fourier Random Feature Representation to create an explicit and learnable mapping function.
  2. Two-Stage Optimization:
    • Phase 1: The model identifies the smallest possible sphere (the "ball") in the feature space that encloses the training data. A reparameterization trick is used to make the kernel parameters learnable via standard gradient descent.
  3. Phase 2: A generator $G(z)$ is trained to ensure that when random noise $z$ is processed, the resulting output—when mapped into the same feature space—falls inside the learned minimal enclosing ball.
  4. Diversity Promotion: To prevent the generator from mapping all noise to a single point (a geometric version of mode collapse), a "feature matching" term is added. This encourages generated samples to spread out and cover the entire surface of the manifold defined by the ball.
Geometric interpretation of GEN: mapping data to a minimal enclosing ball in feature space.

Experimental Setup

GEN was evaluated on a variety of benchmarks to test its robustness and generalization:

  • Synthetic Data: 1D and 2D mixtures of Gaussians and an "S-shape" manifold were used to test mode collapse prevention and manifold learning.
  • MNIST: Handwritten digits (60,000 images) tested with varying subset sizes.
  • Frey Face: Approximately 2,000 images of a single face in various poses.
  • CIFAR-10 & CelebA: Large-scale datasets used to test the model's ability to handle complex, high-dimensional image data.

Key Results

GEN demonstrated several key advantages over traditional density-based models:

  • Anti-Mode Collapse: In synthetic tests where GANs typically collapsed to a single mode, GEN successfully captured all components of Gaussian mixtures.
  • Stability: Quantitative metrics (Symmetric KL and Wasserstein distances) indicated that GEN's training is significantly more stable than GANs, with error rates dropping steadily rather than fluctuating wildly.
  • Data Efficiency: The model produced recognizable MNIST digits and Frey faces even with small training sets (as few as 1,000 images), suggesting a strong geometric prior.
Generated samples from MNIST and Frey Face datasets.

The authors provided mathematical proofs showing that their approximate mapping is bijective. This ensures that the contours learned in the feature space accurately map back to meaningful structures in the original data space. Because the minimal enclosing ball is defined by the entire training set, the generator is geometrically incentivized to cover the whole space, making it naturally resistant to dropping data modes.

Conclusion and Future Work

While GEN successfully captures the geometric structure of complex datasets, the visual crispness of high-resolution images remains a limitation. Fine-grained details in datasets like CelebA are currently not as sharp as those produced by state-of-the-art GANs. Future research may explore more complex geometric shapes beyond simple hyperspheres or refine mapping functions to better capture high-frequency details. GEN establishes a promising new "geometric" branch of generative modeling that avoids the minimax instability of adversarial training.

Summary updated

Abstract

Training model to generate data has increasingly attracted research attention and become important in modern world applications. We propose in this paper a new geometry-based optimization approach to address this problem. Orthogonal to current state-of-the-art density-based approaches, most notably VAE and GAN, we present a fresh new idea that borrows the principle of minimal enclosing ball to train a generator G\left(\bz\right) in such a way that both training and generated data, after being mapped to the feature space, are enclosed in the same sphere. We develop theory to guarantee that the mapping is bijective so that its inverse from feature space to data space results in expressive nonlinear contours to describe the data manifold, hence ensuring data generated are also lying on the data manifold learned from training data. Our model enjoys a nice geometric interpretation, hence termed Geometric Enclosing Networks (GEN), and possesses some key advantages over its rivals, namely simple and easy-to-control optimization formulation, avoidance of mode collapsing and efficiently learn data manifold representation in a completely unsupervised manner. We conducted extensive experiments on synthesis and real-world datasets to illustrate the behaviors, strength and weakness of our proposed GEN, in particular its ability to handle multi-modal data and quality of generated data.
BibTeX
@inproceedings{le2018geometric,
  author = {Trung Le and Hung Vu and Tu Dinh Nguyen and Dinh Q. Phung},
  title = {{Geometric Enclosing Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=BkEyfNzuWS},
}
Figure