SynthCorp: A Platform for Creating Benchmarks for Data Agents

Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon HalevyarXiv 2026 · arXiv

Building an LLM agent that can navigate a simple SQL database is one thing; building one that can handle the messy, evolving reality of a corporate "data estate" is quite another. In a real enterprise, data isn't just a static snapshot; it’s a living history of business events, policy changes, and migrations.

TL;DR

SynthCorp is a platform that generates customizable, complex benchmarks for data agents by simulating business events over a virtual calendar. By recording the full causal provenance of every table update and policy memo, it creates "data estates" that are internally consistent and provides automatically verified "gold programs" for evaluation, exposing reasoning failures in even the most advanced LLMs.

What Problem Does This Paper Address?

Current benchmarks for LLM-driven data agents (like text-to-SQL models) suffer from three major flaws:

  1. Rapid Saturation: Models quickly reach high accuracy on static datasets, making it hard to measure progress.
  2. Lack of Customization: Developers can't easily test specific capabilities, such as an agent's ability to handle redundant tables or resolve ambiguous business terms found only in PDFs.
  3. Missing Context: Most benchmarks provide a static snapshot. In reality, the difficulty of a data task often lies in its history—knowing which table is authoritative after a migration or when a specific accounting policy took effect.

Researchers rarely have access to proprietary corporate data, creating a massive bottleneck in developing agents capable of true enterprise-grade reasoning.

Methods

The core innovation of SynthCorp is shifting from generating data to generating business events. The pipeline consists of four stages:

  1. Event Synthesis: Given a scenario (e.g., "a specialty coffee shop"), an LLM synthesizes routine events (daily sales) and disruption events (a schema upgrade or a new staff discount policy).
  2. Compilation: These events are compiled into programs using a Domain Specific Language (DSL) of twelve Simulation Primitives (SPs). These primitives (like CreateSchema or ApplyTransactions) establish a "boundary of trust," where the LLM handles high-level content generation while the system enforces low-level referential integrity and consistency.
  3. Scheduling & Execution: Events are placed on a virtual calendar. The system executes them in order, materializing a relational database ($D_{db}$), a document store ($D_{doc}$), and a data catalog ($C$). Crucially, it records every operation in a simulation log ($L_{log}$).
  4. Benchmark Generation: The platform uses the simulation log to derive analytical tasks and verified "gold programs" (SQL queries).
The SynthCorp pipeline: from user scenario to verified benchmark.

Datasets

The authors used SynthCorp to generate seven distinct data estates across five domains:

  • Retail: Scaled from 10 to 1,000 operational tables, simulating a coffee shop expanding into a regional chain.
  • Music Store: Based on the Chinook schema but augmented with "canary" accounts and off-database refunds.
  • Banking: Reuses the Czech bank schema from the BIRD benchmark but adds 120 days of simulated history and policy documents.
  • Logistics & Insurance: Focuses on complex reconciliation tasks and unbackfilled migrations.

In total, the initial release includes 256 tasks. Unlike existing benchmarks that rely on human annotation, SynthCorp's tasks are verified against the causal provenance in the simulation log.

Key Insights and Ideas

  • Causal Provenance is the Ground Truth: By knowing exactly how a data point was created (e.g., "this row was written after the Day 3 schema change"), SynthCorp can verify if an agent's query is logically correct without needing a human to manually write the SQL.
  • Cross-Asset Complexity: Real-world reasoning requires looking at more than just the schema. SynthCorp forces agents to find business rules hidden in "policy memos" or "meeting transcripts" within the document store to write the correct query.
  • The "Boundary of Trust": The authors found that LLMs are excellent at describing business processes but struggle to maintain perfect SQL referential integrity over thousands of rows. By using the LLM to generate the logic and the simulation engine to handle the execution, they achieve a near-zero error rate in data generation.

Key Results

The benchmarks effectively separate model tiers and data-access settings:

  • Model Separation: In the most complex settings, Claude Opus 5.5 significantly outperformed DeepSeek V4.1 and Gemma 4, particularly on "composite tasks" that required reasoning across multiple business events.
  • The Necessity of Context: "Schema-only" prompting (S1) failed frequently. Providing even a single retrieved document (S2) raised accuracy by 9 to 17 percentage points.
  • Exposing Reasoning Failures: On a 1,000-table warehouse, human annotators struggled to verify queries. SynthCorp’s log-based audit revealed that state-of-the-art agents often failed to check if "loyalty tiers" had changed after a specific target date, a subtle point-in-time reasoning error that the simulation log caught instantly.

Limitations and Future Work

  • Simulation Boundary: The authors acknowledge that while the metadata complexity is high, the actual data values are stylized. They are not yet simulating deep statistical relationships (e.g., how weather affects coffee sales) unless explicitly requested.
  • Task Difficulty: While challenging for current models, the authors aim to produce even longer chains of business events and deeper multi-document rules to stay ahead of rapidly improving frontier models.
  • Agentic Simulation: Future work includes "LLM agents as employees," where agents themselves drive the business events, creating an even more dynamic and unpredictable evaluation environment.

Summary updated

Abstract

Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass deterministic pipeline checks (such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures) with a near-zero error rate, introduce cross-asset complexities, and cover all requested dimensions. Evaluating three models across four data-context settings shows that the benchmarks separate data contexts and model tiers. Auditing candidate queries against causal provenance in the simulation log verifies gold programs without manual annotation, including on multi-event composite tasks that expose reasoning failures in state-of-the-art agents. We hope the community uses SynthCorp to generate more challenging and useful data estates and benchmarks that foster the development of enterprise data agents.
BibTeX
@misc{zhang2026synthcorp,
  author = {Jiani Zhang and Manav Garg and Eugenie Lai and Cheng Zhen and Eva Sharma and Yang Pan and Yaming Lin and Alon Halevy},
  title = {{SynthCorp: A Platform for Creating Benchmarks for Data Agents}},
  year = {2026},
  eprint = {2611.14765},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14765},
}
Figure