Learning Signal-Agnostic Manifolds of Neural Fields

Yilun Du, Katherine M. Collins, Joshua B. Tenenbaum, Vincent Sitzmann

Introduction

Modern deep learning models are typically specialized for specific data modalities. Generating images requires convolutional architectures, while audio processing often relies on sequence models. This specialization creates "modality silos," complicating tasks that involve irregularly sampled data or cross-modal relationships. Researchers at MIT have introduced GEM (GEnerative Manifold learning), a framework designed to learn the underlying structure of data—be it images, 3D shapes, or audio—using a single, signal-agnostic architecture.

The Challenge of Modality-Dependent Models

Current generative models, such as GANs, often struggle with two primary issues: modality dependence and mode collapse. Modality dependence limits the flexibility of models to handle diverse data types within a unified framework. Mode collapse occurs when a model fails to capture the full variety of a dataset, potentially ignoring significant portions of the data distribution. GEM addresses these by learning a low-dimensional, well-structured manifold of the data that is independent of the signal type, ensuring comprehensive coverage.

Methodology: Neural Fields and Hypernetworks

GEM's architecture rests on two core pillars: the use of neural fields as a universal representation and hypernetworks for manifold mapping.

Neural Fields as a Universal Language

Instead of discrete representations like pixel grids or voxels, GEM represents signals as continuous functions, known as neural fields. A neural field $\Phi$ maps coordinates (e.g., $(x, y)$ for images or $(x, y, z)$ for 3D shapes) to their corresponding values (e.g., color or occupancy). This coordinate-based approach allows the same MLP architecture to represent any signal type.

Hypernetworks for Manifold Mapping

To generate these neural fields, GEM employs a hypernetwork $\Psi$. This "meta-network" takes a low-dimensional latent code $z$ from a learned manifold and predicts the weights for the neural field $\Phi$. This allows the model to navigate a continuous latent space where each point corresponds to a specific signal.

The GEM architecture uses a hypernetwork to map latent codes to the weights of coordinate-based neural fields, enabling signal-agnostic generation across modalities like images and audio.

Geometric Constraints

To ensure the latent space behaves as a proper manifold, GEM incorporates three specific loss functions:

  1. Reconstruction Loss: Ensures every training data point can be accurately recovered, preventing mode collapse.
  2. Local Linearity (LLE) Loss: Encourages the manifold to be locally "flat," facilitating perceptually meaningful interpolations.
  3. Local Isometry Loss: Ensures that distances in the latent space reflect perceptual distances between signals.

Experimental Evaluation

The authors evaluated GEM across four distinct modalities:

  • Images: CelebA-HQ celebrity faces.
  • Audio: NSynth instrument clips, processed as spectrograms.
  • 3D Shapes: ShapeNet objects.
  • Cross-Modal (Audio-Visual): Sub-URMP cello performances, pairing visual frames with audio clips.

Key Results and Insights

GEM demonstrated significant improvements over baselines like StyleGAN2 and Functional Data Networks (FDN).

Superior Reconstruction and Coverage

GEM achieved higher reconstruction accuracy than StyleGAN2 on the CelebA-HQ dataset (MSE of 0.0025 vs. 0.0044). Because GEM uses an auto-decoding approach, it ensures the entire training distribution is represented, avoiding the selective coverage often seen in GANs.

Cross-Modal Completion

A standout capability of GEM is its performance in cross-modal "hallucination." In the cello dataset, the model could generate a correlated audio spectrogram from a single image. For instance, if the image shows a bow high on the strings, GEM generates the corresponding high-pitched audio.

Comparison of cross-modal generation on the cello dataset. GEM successfully generates correlated audio spectrograms from visual input, outperforming StyleGAN2 which produces noisy results.

Perceptual Interpolation and Inpainting

By enforcing local linearity and isometry, GEM enables smooth transitions between different data points (e.g., morphing one face into another). It also supports diverse inpainting; when part of an image is masked, GEM can generate multiple plausible completions that remain contextually consistent.

Discussion and Future Work

While GEM excels at capturing manifold structure and ensuring coverage, it currently lacks the high-frequency "crispness" of specialized high-fidelity models like StyleGAN3. The authors suggest that future work could involve introducing "soft" inductive biases to improve visual quality without sacrificing signal-agnosticism. Additionally, as with all generative models, care must be taken regarding the societal biases present in training data, which GEM can inadvertently learn and propagate.

In conclusion, GEM represents a significant step toward unified generative modeling, demonstrating that geometric constraints on neural fields can effectively organize diverse data modalities into a single, coherent manifold.

Summary updated

Abstract

Deep neural networks have been used widely to learn the latent structure of datasets, across modalities such as images, shapes, and audio signals. However, existing models are generally modality-dependent, requiring custom architectures and objectives to process different classes of signals. We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner. We cast our task as one of learning a manifold, where we aim to infer a low-dimensional, locally linear subspace in which our data resides. By enforcing coverage of the manifold, local linearity, and local isometry, our model -- dubbed GEM -- learns to capture the underlying structure of datasets across modalities. We can then travel along linear regions of our manifold to obtain perceptually consistent interpolations between samples, and can further use GEM to recover points on our manifold and glean not only diverse completions of input images, but cross-modal hallucinations of audio or image signals. Finally, we show that by walking across the underlying manifold of GEM, we may generate new samples in our signal domains.
BibTeX
@inproceedings{du2021learning,
  author = {Yilun Du and Katherine M. Collins and Joshua B. Tenenbaum and Vincent Sitzmann},
  title = {{Learning Signal-Agnostic Manifolds of Neural Fields}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=3rjYr0K-OGC},
}
Figure