Abstract
While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.
Introduction
Large Language Models (LLMs) have revolutionized text generation, yet synthesizing high-quality tabular data remains a significant challenge. Tables present unique difficulties: they contain complex interdependencies between columns, a diverse mix of data types (numerical, categorical, and nested), and lack a natural sequential order. Existing solutions, such as GANs or Diffusion models, often require extensive preprocessing and struggle with non-numerical data. While standard LLMs can be fine-tuned on tables by treating them as flat strings, they often fail to capture the distinct statistical characteristics of individual columns.
Tabby addresses these issues through a clever architectural modification that transforms standard LLMs into high-fidelity generators for tabular and structured data. By replacing the model's output head with a column-specific "Mixture-of-Experts" (MoE) layer, Tabby achieves data quality that reaches parity with real-world datasets, often outperforming models that are orders of magnitude larger.
Methodology
The researchers introduce two primary contributions: the Tabby architecture and the "Plain" training technique.
The Tabby Architecture
Rather than training a new model from scratch, Tabby is implemented as a post-training modification. The standard language modeling head of a pre-trained Transformer (e.g., GPT-2 or Llama) is replaced with a Gated Mixture-of-Experts (MoE) layer. In this configuration, each column in the dataset is assigned its own dedicated set of parameters, or "expert." This allows the model to learn column-specific distributions while leveraging the general knowledge of the pre-trained transformer backbone.
The "Plain" Training Technique
To process tables within the LLM, a simple formatting method is employed. A row is converted into a string following the pattern: "<BOS> v1 is [val1] <EOC> v2 is [val2] ... <EOS>". Despite its simplicity compared to more complex shuffling methods like GReaT, this "Plain" approach proved highly effective when paired with the Tabby architecture.
Experimental Setup and Results
Tabby was evaluated across eight diverse tabular datasets (covering classification and regression tasks) and one nested structured dataset (Glaucoma JSON records). It was compared against traditional tabular generators (CTGAN, TVAE), state-of-the-art diffusion models (TabSyn, TabDiff), and existing LLM-based methods (GReaT, GTT).
Key Results
Tabby reached "upper-bound" performance—matching the utility of real data—on 5 out of 8 tabular datasets. When combining the Tabby architecture with the Plain training technique, the authors observed up to a 44% improvement in data quality over previous state-of-the-art methods.
On the Glaucoma JSON dataset, Tabby achieved an MLE (Machine Learning Efficacy) score of 0.97—identical to the real data—and a near-perfect discrimination score of 0.01, indicating that a classifier could not distinguish synthetic data from real data.
Discussion and Insights
- Architecture Over Scale: A small Tabby model (based on Distilled-GPT2) can outperform massive, non-modified models like Llama-3 8B. For structured data, column-specific parameters appear more critical than raw parameter count.
- Column-Level Tracking: The MoE structure allows researchers to track training loss for each column individually, providing insights into which features are most difficult for the model to master.
- Recursive Structure for JSON: By applying the MoE modification recursively, Tabby inherently understands nested JSON structures, reaching parity with real data where other methods failed.
- Zero-Shot Semantic Understanding: Leveraging text-pretrained LLMs allows Tabby to generate realistic, novel values for high-cardinality categorical features that follow correct semantic patterns, even if those specific strings were not seen during fine-tuning.
Limitations and Future Work
While Tabby shows significant promise, several areas for future research remain. Currently, Tabby lacks formal differential privacy guarantees, which are essential for sensitive domains. Additionally, because Tabby adds parameters for every column, model size scales with table width; future work could explore parameter-sharing techniques to mitigate this. Finally, while more efficient than training from scratch, fine-tuning LLMs still requires more computational resources than lightweight classical methods.
Summary updated