Generative Modeling for Mixed-Type Tabular Data

432 concepts358 papers1 proposal

In computer vision, an image is a homogeneous grid of continuous pixel intensities. In natural language processing, text is a homogeneous sequence of discrete tokens. Deep learning architectures thrive on this homogeneity because it allows them to operate within a uniform mathematical space.

Tabular data, however, is a chaotic marriage of both worlds. A single row in a database might contain a continuous numerical value (e.g., income), a binary flag (e.g., loan default status), and a high-cardinality categorical variable (e.g., zip code). Forcing these disparate data types into a standard neural network creates a fundamental representation gap.

As noted by Remi Domingues et al. in their work on Deep Gaussian Process autoencoders for novelty detection, treating one-hot encoded categorical features as continuous variables during reconstruction severely degrades the quality of learned latent representations. Furthermore, Alfredo Nazábal et al. point out in Robust Variational Autoencoders for Outlier Detection in Mixed-Type Data that comparing likelihoods across mixed types is inherently flawed because continuous and discrete variables operate on entirely disparate probability scales. To build effective generative models for tabular data, researchers must bridge this continuous-discrete divide.

Mode-Specific Normalization for Complex Distributions

Continuous columns in tabular data are rarely normally distributed. They are frequently multi-modal, heavily skewed, or contain discrete "spikes"—such as a financial balance column where exactly $0$ represents a unique logical state rather than just a low value. Standard scaling techniques like MinMax or Z-score normalization flatten these crucial structures, making it difficult for a generator to learn the true underlying distribution.

To solve this, Lei Xu et al. introduced mode-specific normalization in their seminal paper, Modeling Tabular data using Conditional GAN (CTGAN). Instead of treating a continuous column as a single distribution, the authors model it using a Gaussian Mixture Model (GMM).

Raw Continuous Value (e.g., 42.5) 
       │
       ▼ [GMM Fit]
┌────────────────────────────────────────┐
│ Mode 1 (μ=10)  │  Mode 2 (μ=40)        │
│ Weight: 0.1    │  Weight: 0.9          │
└────────────────────────────────────────┘
       │
       ▼ [Transformation]
┌────────────────────────┬───────────────┐
│ One-Hot Vector (Mode)  │ Scalar Offset │
│ [0, 1]                 │ +2.5          │
└────────────────────────┴───────────────┘

Each continuous value is transformed into a representation consisting of two parts: a discrete one-hot vector indicating which Gaussian component (mode) it most likely belongs to, and a continuous scalar representing its specific offset within that component. This effectively transforms a complex, non-normal continuous distribution into a structured, mixed discrete-continuous representation that neural networks can easily process.

Expanding on this concept to handle even messier real-world edge cases, Zilong Zhao et al. developed CTAB-GAN+, which explicitly accounts for mixed-type columns where specific categorical markers and continuous values coexist within the same column, ensuring that unique states (like a null or zero value) are not lost inside a broader continuous distribution.

Unifying Spaces via Latent Embeddings

Instead of engineering the data space column-by-column, an alternative approach maps the entire heterogeneous row into a unified, smooth, continuous latent space. Once the discrete tokens and continuous values are co-embedded into a dense vector space, powerful generative frameworks like diffusion models or flow matching can operate without friction.

This strategy underpins several recent architectures. In Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space (TabSyn), Hengrui Zhang et al. utilize a Transformer-based Variational Autoencoder (VAE) to map mixed-type features into a continuous latent space. A score-based diffusion model then operates entirely within this latent space, capturing complex cross-feature correlations without having to directly manage discrete tokens during the diffusion process. Similarly, the authors of AutoDiff (Namjoon Suh et al.) and TabUnite (Jacob Yoke Hong Si et al.) leverage autoencoders to create a continuous substrate for diffusion and flow matching models.

However, traditional categorical embeddings or one-hot vectors can introduce gradient instabilities when subjected to continuous diffusion noise. To bypass this, Jacob Si et al. proposed a geometric approach called CatConverter in TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation.

Instead of mapping categories to arbitrary high-dimensional vectors, CatConverter maps discrete categorical features onto a 2D unit circle. By embedding categories geometrically as angular coordinates, the authors unify categorical and numerical data into a dense continuous space that stabilizes training, preserves structural relationships, and significantly accelerates generation speeds.

Hybrid and Joint Generative Processes

Rather than forcing data into a shared latent space, some frameworks run separate, specialized generative processes for each data type simultaneously, binding them together within a shared neural backbone.

This is the core mechanism behind TabDDPM: Modelling Tabular Data with Diffusion Models. In this work, Akim Kotelnikov et al. split the generative workload:

  1. Gaussian Diffusion: Applied to continuous numerical features, where noise is added and removed via standard continuous Gaussian transitions.
  2. Multinomial Diffusion: Applied to discrete categorical features, where noise is injected by transitioning tokens uniformly across the category vocabulary based on a discrete transition matrix.

By running these two distinct diffusion processes in parallel and feeding their combined states into a shared network at each denoising step, TabDDPM successfully captures the intricate, non-linear joint distributions across disparate feature types without requiring an explicit autoencoder pre-training step.

Tree-Based Inductive Biases

Deep learning models naturally struggle with the sharp, discontinuous boundaries common in tabular data. Tree-based models (like Random Forests or XGBoost) excel here because they recursively split space rather than trying to fit smooth functions. A parallel branch of research attempts to bring this tree-like inductive bias to generative modeling.

In Adversarial Random Forests for Density Estimation and Generative Modeling, David Watson et al. introduced Adversarial Random Forests (ARFs). ARFs recursively partition data until the features within a given leaf node become locally independent. Once local independence is achieved, generating synthetic data becomes a simple matter of sampling from the empirical marginal distributions within the leaves. This allows for mathematically tractable, lightning-fast synthetic data generation that matches deep learning performance on tabular data while running orders of magnitude faster.

Alternatively, instead of abandoning neural networks, one can inject tree-like inductive biases directly into them. In Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains, Kyungeun Lee et al. proposed a self-supervised pretext task that forces a deep network to predict discretized bin indices rather than raw continuous values.

By training the network to reason about continuous features as discrete intervals, the model learns to handle heterogeneous features and irregular, piecewise-constant functions more effectively. This bridges the representation gap from the opposite direction, giving neural architectures the structural advantages of tree-based models.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heterogeneous_data_integration_heterogen_3dd0eef2,
  title = {heterogeneous data integration (heterogeneous data modeling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-data-integration-heterogeneous-data-modeling/rdr_3dd0eef22d3946789a398737d4fa0ff4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_heterogeneous_data_integration_heterogen_3dd0eef2_v1,
  title = {heterogeneous data integration (heterogeneous data modeling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-data-integration-heterogeneous-data-modeling/rdr_3dd0eef22d3946789a398737d4fa0ff4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake