Copula Flows for Synthetic Data Generation

Sanket Kamthe, Samuel Assefa, Marc Peter DeisenrotharXiv 2021 · OpenReview

TL;DR

This paper introduces "Copula Flows," a novel probabilistic model for generating high-fidelity synthetic data. By combining copula theory with normalizing flows, their method offers improved interpretability and the ability to handle mixed continuous and discrete data, overcoming limitations of existing GAN-based approaches.

What Problem Does This Paper Address?

Generating realistic synthetic data is crucial for scenarios where real data is scarce, sensitive (e.g., medical, financial), or subject to privacy constraints. While Generative Adversarial Networks (GANs) have shown promise, they are often difficult to train, interpret, and struggle with mixed data types (continuous and discrete variables). Existing methods can also be problem-specific, requiring tailored loss functions. This paper aims to provide a more interpretable, flexible, and robust solution for synthetic data generation that can handle complex data distributions, including mixed variable types.

Methods

The core of this work is the "Copula Flow" model, which leverages copula theory and normalizing flows.

  • Copula Theory: This mathematical framework allows for the decomposition of a multivariate distribution into its univariate marginal distributions and a "copula" that captures the dependence structure between variables. This separation is key to understanding and manipulating the data's relationships.
  • Normalizing Flows: These are a class of deep generative models that use a sequence of invertible and differentiable transformations to map a simple base distribution (like a Gaussian or uniform distribution) to a complex target distribution. This allows for exact likelihood computation and efficient sampling.

The Copula Flow model combines these by:

  1. Learning Univariate Marginal Flows: Using neural spline flows (NSFs) to model the distribution of each individual variable. These flows are designed to be invertible, mapping from a uniform distribution to the variable's specific distribution.
  2. Learning the Multivariate Copula Flow: Using another normalizing flow to model the copula, which captures the dependencies between the univariate marginals. This flow transforms independent uniform variables into correlated uniform variables that represent the copula's structure.
  3. Handling Discrete Data: A key innovation is the use of a "distributional transform" to create continuous marginals for discrete variables, allowing the copula framework to be applied seamlessly.

The model is trained by maximizing the data's log-likelihood, with the process of learning marginals often preceding the learning of the copula for better stability.

Datasets

The paper evaluates its method on:

  • Simulated Datasets: Including a "2 rings" dataset to demonstrate the model's ability to capture complex copula structures not easily modeled by standard copulas.
  • Real-world Datasets: The paper mentions using datasets from the UCI Power, Gas, Hepmass, and Miniboone benchmarks for density estimation. For synthetic data generation evaluation, they use datasets from the framework introduced in [Xu et al., 2019], which includes Gaussian mixture (GM) and Bayesian network (BN) simulations, as well as real-world data.

Key Insights and Ideas

  • Copulas as a Bridge to Interpretability: By separating marginal distributions from dependency structures, copulas offer a more interpretable way to understand and generate data compared to end-to-end black-box models like GANs.
  • Normalizing Flows for Flexible Copula Modeling: Normalizing flows provide a powerful, non-parametric tool to learn complex, arbitrary copula structures that go beyond traditional parametric copula families. This allows for modeling intricate dependencies.
  • Unified Framework for Mixed Data: The distributional transform elegantly handles discrete variables, enabling the copula flow model to work with mixed continuous and discrete tabular data, a common challenge in real-world datasets.
  • Universal Density Approximation: The paper theoretically demonstrates that Copula Flows are universal density approximators, meaning they can, in principle, learn any data distribution given sufficient model capacity.

Key Results

  • Density Estimation: The Copula Flow model achieves performance comparable to state-of-the-art neural density estimators on benchmark datasets, as measured by log-likelihood scores. The joint likelihood scores indicate that the model effectively captures relationships between variables.
  • Synthetic Data Generation: When used to generate synthetic data for downstream machine learning tasks (classification and regression), the Copula Flow model performs competitively. For GM simulations, it closely matches the "Identity" transform (using true data), indicating high fidelity. On real data, it performs well on classification and regression tasks, often outperforming or matching existing methods.
  • Complex Copula Learning: The model successfully learns a complex, non-standard copula structure for the "2 rings" dataset, which is difficult for traditional copula methods.
  • Mixed Variable Learning: In an experiment with mixed continuous (Half Normal, Gamma) and discrete (Hypergeometric) variables, the Copula Flow model accurately captures marginals and dependencies, outperforming a mixed vine model in certain aspects, particularly in modeling discrete-to-continuous relationships.

Limitations and Future Work

  • Privacy: While the generated data is synthetic, the paper acknowledges that the current implementation might still be vulnerable to privacy attacks through a large number of queries.
  • Model Complexity: The Copula Flow model uses deep neural networks, which, despite the interpretability offered by the copula framework, can still pose challenges in understanding the exact behavior of the network itself.
  • Future Work: The authors suggest extending the model to incorporate differential privacy guarantees, which would further enhance its utility for sensitive data. They also note that the model's applicability in finance and actuarial science, where copula models are already used, is a promising avenue.

Summary updated

Abstract

The ability to generate high-fidelity synthetic data is crucial when available (real) data is limited or where privacy and data protection standards allow only for limited use of the given data, e.g., in medical and financial data-sets. Current state-of-the-art methods for synthetic data generation are based on generative models, such as Generative Adversarial Networks (GANs). Even though GANs have achieved remarkable results in synthetic data generation, they are often challenging to interpret.Furthermore, GAN-based methods can suffer when used with mixed real and categorical variables.Moreover, loss function (discriminator loss) design itself is problem specific, i.e., the generative model may not be useful for tasks it was not explicitly trained for. In this paper, we propose to use a probabilistic model as a synthetic data generator. Learning the probabilistic model for the data is equivalent to estimating the density of the data. Based on the copula theory, we divide the density estimation task into two parts, i.e., estimating univariate marginals and estimating the multivariate copula density over the univariate marginals. We use normalising flows to learn both the copula density and univariate marginals. We benchmark our method on both simulated and real data-sets in terms of density estimation as well as the ability to generate high-fidelity synthetic data
BibTeX
@misc{kamthe2021copula,
  author = {Sanket Kamthe and Samuel Assefa and Marc Peter Deisenroth},
  title = {{Copula Flows for Synthetic Data Generation}},
  year = {2021},
  eprint = {2101.00598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.00598},
}
Figure