Building an LLM agent that can navigate a simple SQL database is one thing; building one that can handle the messy, evolving reality of a corporate "data estate" is quite another. In a real enterprise, data isn't just a static snapshot; it’s a living history of business events, policy changes, and migrations.
TL;DR
SynthCorp is a platform that generates customizable, complex benchmarks for data agents by simulating business events over a virtual calendar. By recording the full causal provenance of every table update and policy memo, it creates "data estates" that are internally consistent and provides automatically verified "gold programs" for evaluation, exposing reasoning failures in even the most advanced LLMs.
What Problem Does This Paper Address?
Current benchmarks for LLM-driven data agents (like text-to-SQL models) suffer from three major flaws:
- Rapid Saturation: Models quickly reach high accuracy on static datasets, making it hard to measure progress.
- Lack of Customization: Developers can't easily test specific capabilities, such as an agent's ability to handle redundant tables or resolve ambiguous business terms found only in PDFs.
- Missing Context: Most benchmarks provide a static snapshot. In reality, the difficulty of a data task often lies in its history—knowing which table is authoritative after a migration or when a specific accounting policy took effect.
Researchers rarely have access to proprietary corporate data, creating a massive bottleneck in developing agents capable of true enterprise-grade reasoning.
Methods
The core innovation of SynthCorp is shifting from generating data to generating business events. The pipeline consists of four stages:
- Event Synthesis: Given a scenario (e.g., "a specialty coffee shop"), an LLM synthesizes routine events (daily sales) and disruption events (a schema upgrade or a new staff discount policy).
- Compilation: These events are compiled into programs using a Domain Specific Language (DSL) of twelve Simulation Primitives (SPs). These primitives (like
CreateSchemaorApplyTransactions) establish a "boundary of trust," where the LLM handles high-level content generation while the system enforces low-level referential integrity and consistency. - Scheduling & Execution: Events are placed on a virtual calendar. The system executes them in order, materializing a relational database ($D_{db}$), a document store ($D_{doc}$), and a data catalog ($C$). Crucially, it records every operation in a simulation log ($L_{log}$).
- Benchmark Generation: The platform uses the simulation log to derive analytical tasks and verified "gold programs" (SQL queries).
Datasets
The authors used SynthCorp to generate seven distinct data estates across five domains:
- Retail: Scaled from 10 to 1,000 operational tables, simulating a coffee shop expanding into a regional chain.
- Music Store: Based on the Chinook schema but augmented with "canary" accounts and off-database refunds.
- Banking: Reuses the Czech bank schema from the BIRD benchmark but adds 120 days of simulated history and policy documents.
- Logistics & Insurance: Focuses on complex reconciliation tasks and unbackfilled migrations.
In total, the initial release includes 256 tasks. Unlike existing benchmarks that rely on human annotation, SynthCorp's tasks are verified against the causal provenance in the simulation log.
Key Insights and Ideas
- Causal Provenance is the Ground Truth: By knowing exactly how a data point was created (e.g., "this row was written after the Day 3 schema change"), SynthCorp can verify if an agent's query is logically correct without needing a human to manually write the SQL.
- Cross-Asset Complexity: Real-world reasoning requires looking at more than just the schema. SynthCorp forces agents to find business rules hidden in "policy memos" or "meeting transcripts" within the document store to write the correct query.
- The "Boundary of Trust": The authors found that LLMs are excellent at describing business processes but struggle to maintain perfect SQL referential integrity over thousands of rows. By using the LLM to generate the logic and the simulation engine to handle the execution, they achieve a near-zero error rate in data generation.
Key Results
The benchmarks effectively separate model tiers and data-access settings:
- Model Separation: In the most complex settings, Claude Opus 5.5 significantly outperformed DeepSeek V4.1 and Gemma 4, particularly on "composite tasks" that required reasoning across multiple business events.
- The Necessity of Context: "Schema-only" prompting (S1) failed frequently. Providing even a single retrieved document (S2) raised accuracy by 9 to 17 percentage points.
- Exposing Reasoning Failures: On a 1,000-table warehouse, human annotators struggled to verify queries. SynthCorp’s log-based audit revealed that state-of-the-art agents often failed to check if "loyalty tiers" had changed after a specific target date, a subtle point-in-time reasoning error that the simulation log caught instantly.
Limitations and Future Work
- Simulation Boundary: The authors acknowledge that while the metadata complexity is high, the actual data values are stylized. They are not yet simulating deep statistical relationships (e.g., how weather affects coffee sales) unless explicitly requested.
- Task Difficulty: While challenging for current models, the authors aim to produce even longer chains of business events and deeper multi-document rules to stay ahead of rapidly improving frontier models.
- Agentic Simulation: Future work includes "LLM agents as employees," where agents themselves drive the business events, creating an even more dynamic and unpredictable evaluation environment.
Summary updated