TL;DR
SpBench is an "agent-native" framework designed to automate the grueling task of benchmarking diverse data engines (like Snowpark and Apache Spark) within a lakehouse environment. By providing LLM agents with a strictly typed substrate and structured telemetry, the system moves from "equivalence by convention" to "equivalence by construction," reducing benchmark authoring cycles from months to a single automated loop.
What Problem Does This Paper Address?
In the modern cloud data landscape, the same logical workload can often run on multiple execution engines. Comparing these engines fairly is what the authors call a "triple-equivalence problem": you must hold the data, the query semantics, and the compute tier (cost/performance) equal across platforms.
Historically, this has been an expensive, manual process requiring deep domain expertise in multiple platforms. Standard benchmarks like TPC-DS tell you what to run, but not how to implement it identically across different engines or how to verify that the results are truly comparable. "Agent-in-a-shell" approaches—where an LLM simply issues raw commands—frequently fail because they lack the structured context needed to maintain rigor across complex, multi-engine environments.
Methods
SpBench introduces a deterministic "substrate" that serves as the foundation for LLM agents. The framework is built on three core pillars:
-
Strictly Typed Definitions: Instead of writing free-form code, benchmarks are defined using a platform-agnostic hierarchy (BenchmarkDefinition → QueryGroup → Query → QueryStep). This definition is then compiled via templated code generation into engine-specific artifacts, ensuring logical equivalence by design.
Fig. 1. The SpBench substrate. A typed BenchmarkDefinition compiles via templated code generation into per-engine artifacts executed by pluggable runners, emitting star-schema telemetry (fact_runs, fact_queries, dim_*) that the agent surface (Section IV) reads at author- and analysis- time. -
Structured-Telemetry Contract: Every run emits data into a standardized star-schema (fact and dimension tables). This allows agents to query results using deterministic SQL rather than trying to parse messy logs.
- Agent-Native Harness: The framework provides three composable "skills" that agents use to drive the workflow:
- Distill: Ingests production telemetry to create sanitized benchmark specs.
- Build: Transforms specs into runnable code and orchestration wiring.
- Analyze: Queries the telemetry to produce plain-language summaries of performance deltas and regressions.
To manage complexity, the authors emphasize a "Decompose, Don't Monolith" strategy. Rather than one giant agent, the "Build" skill is broken down into specialized sub-agents for code generation, code review, and validation (Scale-Factor-1 testing) before moving to full-scale execution.
Datasets
SpBench was evaluated using a portfolio of over 400 benchmark queries, including:
- Standardized Suites: TPC-DI (data integration) and TPCx-BB (big data analytics).
- Custom Benchmarks: Workloads distilled from real-world production patterns observed on Snowflake.
- Golden Test Case: TeraSort was used as the primary end-to-end test case to ensure reliable execution across all supported engines and configurations.
Key Insights and Ideas
- Equivalence by Construction: The most critical insight is that fairness in benchmarking shouldn't rely on human (or agent) "best effort." By using a typed, code-generated substrate, the framework forces engine-specific implementations to be semantically identical.
- The Importance of "Agent-Facing" Interfaces: LLM productivity depends heavily on the environment. SpBench uses the Model Context Protocol (MCP) to expose tools (like SQL access and artifact retrieval) as typed interfaces, grounding agent interactions in structured calls rather than brittle shell commands.
- Governed Loops: The system operates in two loops: an Operational loop for detecting and classifying regressions, and a Development loop for turning new requests into benchmarks. Both are gated by "staged acceptance contracts"—automated checks that prevent the agent from proceeding if a stage's output is invalid.
Key Results
- Efficiency: The benchmark authoring cycle was reduced from weeks or months to just one AI development loop.
- Scale: The framework currently manages over 400 queries running on a regular cadence.
- Production Impact: SpBench has successfully detected performance regressions in production-scale engines, leading to direct improvements in file-to-table transformations and User-Defined Function (UDF) optimizations.
- Reliability: Through sub-agent delegation and acceptance gates, the system achieved reliable "one-shot" benchmark authoring without manual intervention.
Limitations and Future Work
The authors acknowledge that SpBench is currently a single-organization deployment (Snowflake) targeting a fixed set of engines.
Future directions include: * Broadening Evaluation: Expanding the framework to work across different organizations and a wider variety of execution engines. * Democratizing Resolution: Moving toward a model where any customer-reported performance signal can be automatically resolved by anyone, guided by the AI framework. * Refining Sub-Agent Boundaries: Continuing to identify "natural failure points" in workflows to further isolate and retry agent tasks more efficiently.
Summary updated