Ornstein Auto-Encoders

Youngwon Choi, Joong-Ho WonIJCAI 2019 · DOI

TL;DR

This paper introduces the Ornstein Auto-Encoder (OAE), a novel representation learning model designed for correlated data with nested structures, like images of the same person or different instances of the same digit. By extending the Wasserstein Auto-Encoder to stationary random processes and leveraging exchangeability, OAE can generate realistic variations of both known and, crucially, unknown subjects, a feat previously impossible for conditional generative models.

What Problem Does This Paper Address?

Many real-world datasets, such as collections of images of the same individual or multiple examples of a specific digit, exhibit inherent correlations within groups of data points. This violates the standard "independent and identically distributed" (i.i.d.) assumption common in machine learning, making it challenging for traditional models to learn effective representations. Existing methods often struggle with large numbers of categories (e.g., thousands of identities) or data imbalance, and conditional generative models like Conditional Adversarial Auto-Encoders (CAAE) cannot generate samples for entirely new, unseen subjects. The paper aims to address these limitations by developing a generative model that explicitly accounts for the nested, correlated structure of data.

Methods

The core contribution is the Ornstein Auto-Encoder (OAE). OAE builds upon the Wasserstein Auto-Encoder (WAE) by extending the concept of optimal transport distance to stationary random processes. This is achieved by introducing Ornstein's d-bar distance, an optimal transport distance between random processes.

Key methodological points include:

  • Extension of WAE to Processes: The paper shows that a theorem by Bousquet et al. (2017) for WAEs can be extended to stationary random processes.
  • Exchangeability Assumption: A crucial simplification arises when assuming exchangeability within a process (meaning the order of observations within a group doesn't matter, which holds for datasets like VGGFace2 and MNIST). This reduces the OAE's optimization problem to a form very similar to WAE, making it tractable.
  • Latent Space Structure: OAE models the latent space by introducing a random variable (e.g., B) that captures the group-level information (like identity), and then conditions the latent variables (Z) on this group variable. This allows for disentangled representations where subjects are clustered.
  • Generative Process: The OAE learns an encoder that maps data to this structured latent space and a decoder that maps from the latent space back to the data space. The decoder is designed to map a sequence of latent variables to a sequence of observations.

Datasets

  • Toy Model: A custom-generated dataset was used to test the model's ability to learn a known low-dimensional distribution embedded in a higher dimension.
  • VGGFace2: A large-scale face dataset with many identities and significant data imbalance. Used for evaluating the generation of known and unknown subjects.
  • MNIST: A standard dataset of handwritten digits. Used to evaluate performance in both balanced and imbalanced settings, particularly for generating class-conditional samples.

Key Insights and Ideas

  • Nesting Matters: The paper highlights that explicitly modeling the "nested structure" of data (e.g., multiple images per person) is key to improving generative capabilities.
  • Ornstein's d-bar Distance: This novel distance metric for random processes is the theoretical foundation for extending optimal transport to this domain.
  • Exchangeability Simplifies: The assumption of exchangeability is critical for making the OAE computationally feasible, bridging the gap between theoretical process distances and practical deep learning optimization.
  • Generating Unknowns is Possible: The most significant insight is that OAE can generate realistic variations for subjects not seen during training. This is a major advantage over conditional generative models that require explicit conditioning on known classes.
  • Disentangled Representations: The structured latent space learned by OAE naturally leads to subject-level disentanglement, where different identities are well-separated in the latent representation.

Key Results

  • VGGFace2: OAE generated higher quality variations for known identities compared to CAAE. Crucially, OAE could generate new variations for unknown subjects with sample quality comparable to WAE (which generates random identities), a capability not possible with CAAE.
  • Vector Arithmetic: The random intercept model in the encoder allows for meaningful vector arithmetic in the latent space, enabling operations like "switching identities" between generated images.
  • MNIST: OAE achieved higher accuracy in generating class-conditional digits than CAAE, especially in imbalanced settings. The reconstruction quality was slightly higher for OAE, but this did not negatively impact classifier performance.
  • Toy Model: OAE successfully learned the underlying structure of a custom toy dataset, demonstrating its ability to capture complex correlations.

Limitations and Future Work

  • Infinite Sequence Mapping: The paper notes that the OAE decoder needs to map an infinite sequence to another infinite sequence, which can pose computational challenges.
  • Divergence Computation: Computing divergences between process distributions can be complex.
  • Future Work: The authors plan to extend OAE to non-exchangeable stationary random processes and explore latent variable modeling of multilevel data, which are common in biomedical applications.

Summary updated

BibTeX
@inproceedings{choi2019ornstein,
  author = {Youngwon Choi and Joong-Ho Won},
  title = {{Ornstein Auto-Encoders}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/301},
}

Loading related content...

Figure