SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking

Lyndon Shi, Nic Crouch, Yi Hu, Zihao Ye, Choden Konigsmark, Eric Vandenberg, Dionysios Logothetis, Kuba Niemiec, Krzysztof Gadomski, Yuxiong HearXiv 2026 · arXiv

TL;DR

SpBench is an "agent-native" framework designed to automate the grueling task of benchmarking diverse data engines (like Snowpark and Apache Spark) within a lakehouse environment. By providing LLM agents with a strictly typed substrate and structured telemetry, the system moves from "equivalence by convention" to "equivalence by construction," reducing benchmark authoring cycles from months to a single automated loop.

What Problem Does This Paper Address?

In the modern cloud data landscape, the same logical workload can often run on multiple execution engines. Comparing these engines fairly is what the authors call a "triple-equivalence problem": you must hold the data, the query semantics, and the compute tier (cost/performance) equal across platforms.

Historically, this has been an expensive, manual process requiring deep domain expertise in multiple platforms. Standard benchmarks like TPC-DS tell you what to run, but not how to implement it identically across different engines or how to verify that the results are truly comparable. "Agent-in-a-shell" approaches—where an LLM simply issues raw commands—frequently fail because they lack the structured context needed to maintain rigor across complex, multi-engine environments.

Methods

SpBench introduces a deterministic "substrate" that serves as the foundation for LLM agents. The framework is built on three core pillars:

  1. Strictly Typed Definitions: Instead of writing free-form code, benchmarks are defined using a platform-agnostic hierarchy (BenchmarkDefinition → QueryGroup → Query → QueryStep). This definition is then compiled via templated code generation into engine-specific artifacts, ensuring logical equivalence by design.

    Fig. 1. The SpBench substrate. A typed BenchmarkDefinition compiles via templated code generation into per-engine artifacts executed by pluggable runners, emitting star-schema telemetry (fact_runs, fact_queries, dim_*) that the agent surface (Section IV) reads at author- and analysis- time.
  2. Structured-Telemetry Contract: Every run emits data into a standardized star-schema (fact and dimension tables). This allows agents to query results using deterministic SQL rather than trying to parse messy logs.

  3. Agent-Native Harness: The framework provides three composable "skills" that agents use to drive the workflow:
  4. Distill: Ingests production telemetry to create sanitized benchmark specs.
  5. Build: Transforms specs into runnable code and orchestration wiring.
  6. Analyze: Queries the telemetry to produce plain-language summaries of performance deltas and regressions.

To manage complexity, the authors emphasize a "Decompose, Don't Monolith" strategy. Rather than one giant agent, the "Build" skill is broken down into specialized sub-agents for code generation, code review, and validation (Scale-Factor-1 testing) before moving to full-scale execution.

Fig. 4. Decomposed Build skill architecture. Authoring is delegated at four points — static codegen, code review, SF1 validation, full-scale run — each handled by narrow-scope sub-agents (eng: per-engine; dir: per-query- directory).

Datasets

SpBench was evaluated using a portfolio of over 400 benchmark queries, including:

  • Standardized Suites: TPC-DI (data integration) and TPCx-BB (big data analytics).
  • Custom Benchmarks: Workloads distilled from real-world production patterns observed on Snowflake.
  • Golden Test Case: TeraSort was used as the primary end-to-end test case to ensure reliable execution across all supported engines and configurations.

Key Insights and Ideas

  • Equivalence by Construction: The most critical insight is that fairness in benchmarking shouldn't rely on human (or agent) "best effort." By using a typed, code-generated substrate, the framework forces engine-specific implementations to be semantically identical.
  • The Importance of "Agent-Facing" Interfaces: LLM productivity depends heavily on the environment. SpBench uses the Model Context Protocol (MCP) to expose tools (like SQL access and artifact retrieval) as typed interfaces, grounding agent interactions in structured calls rather than brittle shell commands.
  • Governed Loops: The system operates in two loops: an Operational loop for detecting and classifying regressions, and a Development loop for turning new requests into benchmarks. Both are gated by "staged acceptance contracts"—automated checks that prevent the agent from proceeding if a stage's output is invalid.
Fig. 3. SpBench’s two governed harness loops. The operational loop repro- duces, localizes, and classifies a telemetry-detected regression; the develop- ment loop turns an external request into a spec, code, and a results summary. Both loops are gated by staged acceptance contracts before completion.

Key Results

  • Efficiency: The benchmark authoring cycle was reduced from weeks or months to just one AI development loop.
  • Scale: The framework currently manages over 400 queries running on a regular cadence.
  • Production Impact: SpBench has successfully detected performance regressions in production-scale engines, leading to direct improvements in file-to-table transformations and User-Defined Function (UDF) optimizations.
  • Reliability: Through sub-agent delegation and acceptance gates, the system achieved reliable "one-shot" benchmark authoring without manual intervention.

Limitations and Future Work

The authors acknowledge that SpBench is currently a single-organization deployment (Snowflake) targeting a fixed set of engines.

Future directions include: * Broadening Evaluation: Expanding the framework to work across different organizations and a wider variety of execution engines. * Democratizing Resolution: Moving toward a model where any customer-reported performance signal can be automatically resolved by anyone, guided by the AI framework. * Refining Sub-Agent Boundaries: Continuing to identify "natural failure points" in workflows to further isolate and retry agent tasks more efficiently.

Summary updated

Abstract

SpBench is an agent-native framework for cross-engine lakehouse benchmarking, in production use for benchmarking Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standardized suites -- such as TPCx-BB and TPC-DI -- and custom benchmarks distilled from production workload patterns. It runs continuously across a broad portfolio of workloads, guiding the performance roadmap and detecting regressions for production-scale data engines. Benchmarking has historically been an expensive investment requiring continuous cross-functional collaboration and domain expertise, making it difficult to obtain actionable performance insights. SpBench provides (i) a typed, code-generated benchmark substrate that compiles platform-agnostic benchmark definitions into fair, comparable platform-specific implementations; (ii) a structured-telemetry contract recording every run as queryable fact and dimension data, letting LLM agents query results deterministically; and (iii) an agent-native harness of three composable skills to expedite building benchmarks and analyzing results. This agent-native architecture enabled the rapid development of more than 400 benchmark queries that run on a regular cadence to detect performance regressions; catching these has directly led to improvements in production.
BibTeX
@misc{shi2026spbench,
  author = {Lyndon Shi and Nic Crouch and Yi Hu and Zihao Ye and Choden Konigsmark and Eric Vandenberg and Dionysios Logothetis and Kuba Niemiec and Krzysztof Gadomski and Yuxiong He},
  title = {{SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking}},
  year = {2026},
  eprint = {2611.00449},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00449},
}
Figure