DEREC-SIMPRO: unlock Language Model benefits to advance Synthesis in Data Clean Room

Tung Sum Thomas Kwok, Chi-Hua Wang, Guang ChengarXiv 2024 · OpenReview

Introduction

Data collaboration in Data Clean Rooms (DCRs)—environments where parties like advertisers and publishers share data without exposing raw sensitive information—is often hindered by complex data structures. While synthetic data serves as a privacy-preserving "mathematical twin" of real data, current generators struggle with many-to-many relationships common in collaborative datasets.

This paper introduces DEREC-SIMPRO, a framework designed to restructure multi-table data for advanced synthesizers and evaluate the results using rigorous, shape-sensitive metrics.

The Challenge: Architectural Mismatch and Evaluation Bias

The authors identify two primary roadblocks in synthetic data generation for DCRs:

  1. Architectural Mismatch: Advanced multi-table synthesizers, particularly those leveraging Large Language Models (LLMs), typically require a strict one-to-many (parent-child) hierarchy. Real-world collaborative data is often many-to-many, leading to poor synthesis quality.
  2. Evaluation Bias: Standard industry metrics (e.g., SDMetrics) often favor "smoothed" distributions. This can result in high scores for models that miss the unique, jagged "shape" of the original data, failing to capture true statistical utility.

Methodology: The DEREC-SIMPRO Framework

The proposed solution consists of two integrated components: the DEREC pipeline for data restructuring and the SIMPRO framework for evaluation.

The DEREC-SIMPRO pipeline for synthetic data generation and evaluation in Data Clean Rooms.

1. The DEREC Pipeline (DEtect, REcreate, Connect)

DEREC acts as a pre-processing bridge to transform incompatible many-to-many data into a synthesizer-friendly one-to-many format:

  • DEtect: Automatically identifies "contextual" columns (e.g., age, gender) that remain consistent for a user.
  • REcreate: Constructs an artificial "parent table" where each subject appears once, populated by contextual columns.
  • Connect: Links the new parent table to the original tables (now "child tables"), establishing the hierarchy required by LLM-based synthesizers.

2. The SIMPRO Evaluation Framework

SIMPRO addresses "smoothing bias" by evaluating data across three dimensions:

  • Statistical Similarity: Uses the Kolmogorov-Smirnov (KS) test to ensure synthetic data captures overall table logic.
  • IMprovement Counts: Provides a granular analysis of how many specific column pairs improved compared to baseline models.
  • PRObabilistic Distance: Employs Wasserstein Distance to measure the literal "cost" of transforming synthetic distributions into real ones.

Key Insights and Results

The study utilized the CTR Prediction - 2022 DIGIX Global AI Challenge dataset, a real-world marketing dataset with complex relational structures.

Engineering Hierarchy

A key insight is that natural hierarchies are not a prerequisite for multi-table synthesizers. By algorithmically "inventing" a parent table through DEREC, the power of LLMs (like GPT-2) can be unlocked for previously incompatible datasets.

Shape Fidelity over Smoothness

The authors demonstrate that capturing the "true" distribution shape is more critical than achieving a smooth curve. SIMPRO prioritizes this "shape fidelity," identifying cases where traditional metrics might reward inaccurate but smooth distributions.

Comparison of real vs. synthetic data distributions, highlighting the framework's ability to capture complex distribution shapes.

Performance Gains

  • Fidelity Boost: The DEREC pipeline consistently improved synthetic data quality. In tests, DEREC-REaLTabFormer showed 147 improved column pairs compared to only 32 that worsened.
  • Superiority over Baselines: The DEREC-enhanced model significantly outperformed single-table benchmarks like CT-GAN and TabDDPM, which failed to capture the collaborative data's underlying logic.

Limitations and Future Work

While effective, the current DEREC implementation synthesizes parent-child pairs independently, potentially missing correlations between different child tables. Future work includes developing a unified training approach and exploring more modern LLM backbones like GPT-4o or Llama 3 to further enhance synthesis quality.

Summary updated

Abstract

Data collaboration via Data Clean Room offers value but raises privacy concerns, which can be addressed through synthetic data and multi-table synthesizers. Common multi-table synthesizers fail to perform when subjects occur repeatedly in both tables. This is an urgent yet unresolved problem, since having both tables with repeating subjects is common. To improve performance in this scenario, we present the DEREC 3-step pre-processing pipeline to generalize adaptability of multi-table synthesizers. We also introduce the SIMPRO 3-aspect evaluation metrics, which leverage conditional distribution and large-scale simultaneous hypothesis testing to provide comprehensive feedback on synthetic data fidelity at both column and table levels. Results show that using DEREC improves fidelity, and multi-table synthesizers outperform single-table counterparts in collaboration settings. Together, the DEREC-SIMPRO pipeline offers a robust solution for generalizing data collaboration, promoting a more efficient, data-driven society.
BibTeX
@misc{kwok2024derec,
  author = {Tung Sum Thomas Kwok and Chi-Hua Wang and Guang Cheng},
  title = {{DEREC-SIMPRO: unlock Language Model benefits to advance Synthesis in Data Clean Room}},
  year = {2024},
  eprint = {2411.00879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00879},
}
Figure