Tabby: Tabular Data Synthesis with Language Models

Sonia Cromp, Satya Sai Srinath Namburi Gnvv, Mohammed Alkhudhayri, Catherine Cao, Samuel Guo, Nicholas Carl Roberts, Frederic Sala

Abstract

While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.

Introduction

Large Language Models (LLMs) have revolutionized text generation, yet synthesizing high-quality tabular data remains a significant challenge. Tables present unique difficulties: they contain complex interdependencies between columns, a diverse mix of data types (numerical, categorical, and nested), and lack a natural sequential order. Existing solutions, such as GANs or Diffusion models, often require extensive preprocessing and struggle with non-numerical data. While standard LLMs can be fine-tuned on tables by treating them as flat strings, they often fail to capture the distinct statistical characteristics of individual columns.

Tabby addresses these issues through a clever architectural modification that transforms standard LLMs into high-fidelity generators for tabular and structured data. By replacing the model's output head with a column-specific "Mixture-of-Experts" (MoE) layer, Tabby achieves data quality that reaches parity with real-world datasets, often outperforming models that are orders of magnitude larger.

Methodology

The researchers introduce two primary contributions: the Tabby architecture and the "Plain" training technique.

The Tabby Architecture

Rather than training a new model from scratch, Tabby is implemented as a post-training modification. The standard language modeling head of a pre-trained Transformer (e.g., GPT-2 or Llama) is replaced with a Gated Mixture-of-Experts (MoE) layer. In this configuration, each column in the dataset is assigned its own dedicated set of parameters, or "expert." This allows the model to learn column-specific distributions while leveraging the general knowledge of the pre-trained transformer backbone.

Comparison of standard LLM architecture versus the Tabby Multi-Head version with column-specific experts.

The "Plain" Training Technique

To process tables within the LLM, a simple formatting method is employed. A row is converted into a string following the pattern: "<BOS> v1 is [val1] <EOC> v2 is [val2] ... <EOS>". Despite its simplicity compared to more complex shuffling methods like GReaT, this "Plain" approach proved highly effective when paired with the Tabby architecture.

Experimental Setup and Results

Tabby was evaluated across eight diverse tabular datasets (covering classification and regression tasks) and one nested structured dataset (Glaucoma JSON records). It was compared against traditional tabular generators (CTGAN, TVAE), state-of-the-art diffusion models (TabSyn, TabDiff), and existing LLM-based methods (GReaT, GTT).

Key Results

Tabby reached "upper-bound" performance—matching the utility of real data—on 5 out of 8 tabular datasets. When combining the Tabby architecture with the Plain training technique, the authors observed up to a 44% improvement in data quality over previous state-of-the-art methods.

Performance profile showing the fraction of problems solved within a performance ratio tau, highlighting Tabby's superior performance across various benchmarks.

On the Glaucoma JSON dataset, Tabby achieved an MLE (Machine Learning Efficacy) score of 0.97—identical to the real data—and a near-perfect discrimination score of 0.01, indicating that a classifier could not distinguish synthetic data from real data.

Discussion and Insights

  • Architecture Over Scale: A small Tabby model (based on Distilled-GPT2) can outperform massive, non-modified models like Llama-3 8B. For structured data, column-specific parameters appear more critical than raw parameter count.
  • Column-Level Tracking: The MoE structure allows researchers to track training loss for each column individually, providing insights into which features are most difficult for the model to master.
  • Recursive Structure for JSON: By applying the MoE modification recursively, Tabby inherently understands nested JSON structures, reaching parity with real data where other methods failed.
  • Zero-Shot Semantic Understanding: Leveraging text-pretrained LLMs allows Tabby to generate realistic, novel values for high-cardinality categorical features that follow correct semantic patterns, even if those specific strings were not seen during fine-tuning.

Limitations and Future Work

While Tabby shows significant promise, several areas for future research remain. Currently, Tabby lacks formal differential privacy guarantees, which are essential for sensitive domains. Additionally, because Tabby adds parameters for every column, model size scales with table width; future work could explore parameter-sharing techniques to mitigate this. Finally, while more efficient than training from scratch, fine-tuning LLMs still requires more computational resources than lightweight classical methods.

Summary updated

Abstract

While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.
BibTeX
@article{cromp2025tabby,
  author = {Sonia Cromp and Satya Sai Srinath Namburi GNVV and Mohammed Alkhudhayri and Catherine Cao and Samuel Guo and Nicholas Carl Roberts and Frederic Sala},
  title = {{Tabby: Tabular Data Synthesis with Language Models}},
  journal = {Accepted by TMLR},
  year = {2025},
  doi = {10.48550/arxiv.2503.02152},
}
Figure