A Systematic Evaluation of Generative Models on Tabular Transportation Data

Chengen Wang, Alvaro A. Cárdenas, Gurcan Comert, Murat Kantarcioglu

Introduction

The proliferation of large-scale transportation datasets, such as taxi and ride-sharing records, offers unprecedented opportunities for urban planning and infrastructure optimization. However, the release of such data is fraught with privacy risks. Mobility data is notoriously re-identifiable; even a small number of known spatiotemporal points can reveal sensitive information about an individual's home, workplace, and daily routines.

Synthetic data generation has emerged as a potential solution to this privacy-utility trade-off. By training generative models to learn the underlying distribution of real-world data, researchers can create synthetic datasets that preserve statistical properties without exposing individual records. While many generative models exist for general tabular data, transportation data presents a unique challenge: it is inherently relational. A single trip is not just a row in a table; it represents an edge in a complex urban transportation network. This paper provides a rigorous evaluation of state-of-the-art generative models—including GANs, VAEs, and Diffusion models—to determine how well they handle the structural and privacy requirements of mobility data.

Methodology and Benchmarked Models

The study benchmarks six prominent generative architectures across three distinct families:

  • Statistical Models: Gaussian Copula, serving as a baseline for traditional statistical synthesis.
  • Generative Adversarial Networks (GANs): CTGAN and CTABGAN, which are specifically designed to handle the mixed-type distributions found in tabular data.
  • Variational Autoencoders (VAEs): TVAE, which uses a neural network to learn a latent representation of the data.
  • Diffusion Models: TabDDPM and STaSy, representing the current state-of-the-art in generative modeling.

Novel Evaluation Metrics

To move beyond standard statistical comparisons, the authors introduce two critical evaluation tools designed for the transportation domain:

  1. Graph Similarity Metric: This metric treats pickup and drop-off zones as nodes and trips as edges. By measuring the Total Variation Distance between the trip distributions of real and synthetic networks, the authors can quantify how well a model preserves the city's actual traffic flow and network topology.
  2. Distance to Closest Record Ratio (rDCR): Traditional privacy metrics often fail to detect subtle overfitting. The rDCR measures the ratio of the distance to the training data versus the distance to holdout (testing) data. A ratio significantly below 1.0 indicates that the model is "leaking" specific training examples, making it vulnerable to membership inference attacks.

Experimental Setup

The evaluation utilized the 2015 Green Taxi Trip Data from New York City. The original dataset of 19.2 million rows was sampled down to 40,000 records for training and 20,000 for testing. The data is highly complex, featuring 22 columns of mixed types, including categorical variables (zones), integers (passenger counts), and floats (trip distances and fares). To facilitate the graph-based analysis, the authors mapped raw coordinates to specific NYC taxi zones using 2019 reference data.

Key Insights and Results

The Utility of Diffusion Models

Diffusion-based models, particularly TabDDPM, emerged as the leaders in data utility. TabDDPM achieved an $R^2$ of 94.69 for downstream regression tasks (predicting total trip amounts), significantly outperforming GAN-based models like CTGAN, which reached only ~80.5. Furthermore, TabDDPM demonstrated the highest "coverage" (68.69%), indicating it captured a broader portion of the real data distribution than its competitors.

The "Graph Gap"

Despite their statistical prowess, all models struggled to replicate the underlying transportation network. While TVAE and CTABGAN performed relatively better on the Graph Similarity Metric, there remains a significant performance gap between synthetic and real-world networks. Diffusion models like TabDDPM actually performed poorly here, suffering from "mode collapse" on rare zone-to-zone pairs. This suggests that while these models are excellent at capturing general column distributions, they struggle with the high-cardinality categorical variables that define the network structure.

Privacy Vulnerabilities

The rDCR metric revealed that average privacy scores can be misleading. While many models appeared safe on average, the authors found that models like the Gaussian Copula were highly vulnerable at the "tails" of the distribution. The 5% most unique records in the synthetic data were often too close to real training records, potentially allowing an adversary to confirm the presence of specific individuals in the original dataset.

Computational Efficiency

The study also highlighted a trade-off between quality and speed. The Gaussian Copula was nearly instantaneous but lacked utility. Conversely, CTABGAN and STaSy were the most computationally expensive, with training times exceeding 100 minutes for the relatively small sample size used in the study.

Conclusion

While diffusion models currently represent the frontier for synthetic tabular data utility, they are not yet a "plug-and-play" solution for transportation data. The difficulty in handling high-cardinality variables and the failure to fully capture graph-based relationships indicate that domain-specific constraints must be integrated into future generative architectures. Future research should focus on developing models that explicitly incorporate graph topology and improving hyperparameter tuning to prevent mode collapse in sparse mobility datasets.

Summary updated

BibTeX
@inproceedings{wang2025systematic,
  author = {Chengen Wang and Alvaro A. Cárdenas and Gurcan Comert and Murat Kantarcioglu},
  title = {{A Systematic Evaluation of Generative Models on Tabular Transportation Data}},
  booktitle = {PAKDD},
  year = {2025},
  doi = {10.1007/978-981-96-8298-0\_9},
}
Figure