TabDLM: Free-Form Tabular Data Generation via Joint Numerical-Language Diffusion

Donghong Cai, Jiarui Feng, Yanbo Wang, Da Zheng, Yixin Chen, Muhan Zhang

TL;DR

TabDLM is a new framework that generates high-fidelity synthetic tabular data containing a mix of numbers, categories, and free-form text. By combining continuous numerical diffusion with discrete masked language diffusion in a single bidirectional model, it captures complex cross-modal dependencies that traditional LLMs and diffusion models often miss.

What Problem Does This Paper Address?

Synthetic tabular data is essential for data augmentation and privacy-preserving machine learning. However, modern datasets are increasingly "messy," containing structured attributes alongside free-form text (like product reviews or clinical notes).

Existing tools generally fall into two camps, both with significant flaws:

  1. Diffusion-based methods: Excellent at modeling numerical correlations, but they struggle with the "exponential size" of natural language, often producing garbled or low-quality text.
  2. LLM-based methods: Great at generating fluent text, but they treat numbers as strings of discrete tokens (e.g., "123.45" might be split into three separate tokens). This fragmentation makes it difficult for the model to understand numerical precision or wide ranges, leading to poor statistical accuracy.

Furthermore, standard LLMs are autoregressive (generating left-to-right), which is a poor fit for tables where columns often have mutual, non-linear dependencies.

Methods

The authors introduce TabDLM, which uses a Masked Diffusion Language Model (MDLM) as its backbone. Unlike GPT-style models, MDLMs use bidirectional attention, allowing every column to "see" every other column during the generation process.

Key technical innovations include:

  • Joint Numerical–Language Diffusion: The model handles two types of diffusion simultaneously. It uses discrete masked diffusion for text and categories (filling in "blanks") and continuous diffusion for numerical values.
  • Specialized Number Encoding: Instead of tokenizing numbers into digits, TabDLM uses a trainable MLP-based encoder to map each numerical value into a single $d$-dimensional embedding. This preserves the "number-ness" of the data.
  • Unified Noise Schedule: The model uses a single diffusion time step ($t$) to synchronize the denoising of both text and numbers, ensuring the generated text is grounded in the generated numerical facts.
  • Efficient Fine-tuning: The system is built on top of a large pre-trained model (LLaDA-8B) and uses Parameter-Efficient Fine-Tuning (LoRA) to adapt to specific datasets.

Datasets

The researchers evaluated TabDLM on several fronts:

  • New Synthetic Benchmarks: They introduced MathExpr (numbers + operators + LaTeX expressions) and ProfileBio (demographics + natural language biographies) to specifically test if the model can maintain consistency between structured data and text.
  • Real-World Heterogeneous Data: The Amazon dataset (product metadata + reviews) was used to test performance on complex, real-world distributions.
  • Standard Tabular Benchmarks: To ensure it remains a "good citizen" in traditional tasks, they tested on five classic datasets without text: Adult, Default, Shoppers, Magic, and Beijing.

Key Insights and Ideas

  • Bidirectional Attention is the "Secret Sauce": Because tables aren't naturally sequential, the ability of a Masked LM to look at all columns at once is far superior to the left-to-right generation of standard LLMs for capturing inter-column correlations.
  • Continuous vs. Discrete: The paper argues that we shouldn't force numbers into a discrete language space. By keeping numbers in a continuous space and text in a discrete space—but denoising them in the same transformer layers—we get the best of both worlds.
  • Semantic Grounding: The results suggest that TabDLM doesn't just generate "plausible" text; it generates text that is semantically grounded in the accompanying numerical data (e.g., if the "Age" column says 22, the biography correctly describes the person as being in the "early stage of adulthood").

Key Results

  • Superior Correlation Tracking: On the MathExpr dataset, TabDLM reduced the "Trend" error (a measure of how well correlations are preserved) by 77.6% compared to the strongest LLM baseline.
  • High-Fidelity Text: On the Amazon dataset, TabDLM achieved a Shape Error of 4.67% and a Trend Error of 5.33%, significantly outperforming LLaDA-8B and Qwen-based baselines.
  • General Purpose Strength: Even on "classic" numerical-only datasets, TabDLM outperformed or matched state-of-the-art specialized diffusion models like TabDiff, proving it doesn't sacrifice structured data quality to gain text capabilities.

Limitations and Future Work

  • Inference Speed: Because it relies on a large 8B-parameter backbone and an iterative diffusion process, TabDLM is slower to sample from than smaller, specialized tabular models.
  • Shared Noise Schedules: Currently, the model uses the same noise level for both text and numbers. The authors suggest that "modality-specific" noise schedules might allow for even finer control and better quality in future iterations.
  • Computational Cost: Training and running an 8B model requires significant GPU resources (the study used NVIDIA A100s), which might be a barrier for some practitioners.

Summary updated

Abstract

Synthetic tabular data generation has attracted growing attention due to its importance for data augmentation, foundation models, and privacy. However, real-world tabular datasets increasingly contain free-form text fields (e.g., reviews or clinical notes) alongside structured numerical and categorical attributes. Generating such heterogeneous tables with joint modeling of different modalities remains challenging. Existing approaches broadly fall into two categories: diffusion-based methods and LLM-based methods. Diffusion models can capture complex dependencies over numerical and categorical features in continuous or discrete spaces, but extending them to open-ended text is nontrivial and often leads to degraded text quality. In contrast, LLM-based generators naturally produce fluent text, yet their discrete tokenization can distort precise or wide-range numerical values, hindering accurate modeling of both numbers and language. In this work, we propose TabDLM, a unified framework for free-form tabular data generation via a joint numerical--language diffusion model built on masked diffusion language models (MDLMs). TabDLM models textual and categorical features through masked diffusion, while modeling numerical features with a continuous diffusion process through learned specialized numeric tokens embedding; bidirectional attention then captures cross-modality interactions within a single model. Extensive experiments on diverse benchmarks demonstrate the effectiveness of TabDLM compared to strong diffusion- and LLM-based baselines.
BibTeX
@misc{cai2026tabdlm,
  author = {Donghong Cai and Jiarui Feng and Yanbo Wang and Da Zheng and Yixin Chen and Muhan Zhang},
  title = {{TabDLM: Free-Form Tabular Data Generation via Joint Numerical-Language Diffusion}},
  year = {2026},
  eprint = {2602.22586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.22586},
}
Figure