TL;DR
FlinkEval is a new, execution-grounded benchmark designed to evaluate how well Large Language Models (LLMs) translate natural language into streaming SQL (Apache Flink). Unlike previous benchmarks that focus on static data, FlinkEval uses a live cluster to validate queries against complex streaming semantics like event time and watermarks, revealing that even frontier models struggle significantly as query complexity increases or user intent becomes underspecified.
What Problem Does This Paper Address?
Most Natural Language to SQL (NL2SQL) research relies on benchmarks like Spider or BIRD, which primarily deal with "bounded" relational data—think of a static Excel sheet or a traditional database. However, modern data engineering often relies on streaming engines like Apache Flink, which process continuous, evolving data streams.
Streaming SQL is a different beast. Success depends on more than just picking the right columns; the query must respect temporal relationships, watermarks, and changelog behaviors. For instance, a query that works on a static table might fail in a stream if it tries to perform a pattern match on data that produces updates rather than append-only events. FlinkEval addresses the lack of evaluation frameworks for these operational and temporal constraints.
Methods
The authors built FlinkEval using an iterative, "execution-grounded" approach:
- Synthetic Generation: They created 50 domain schemas and temporally consistent data across 45 industry domains.
- Pattern-Driven SQL: They collected 46 SQL patterns from expert-written pairs and vendor documentation, covering complex features like windowing, temporal joins, and
MATCH_RECOGNIZE. - Live Validation & Repair Loop: Every generated "gold" query was executed on a live Confluent Cloud Apache Flink cluster. If a query failed or returned no data, it entered a repair loop until it was operationally valid.
- The Question Specification Ladder: To simulate real-world ambiguity, they developed a seven-level ladder of natural language requests, ranging from "Schema-Explicit" (technical and precise) to "Unclarified Underspecified" (vague business language).
Evaluation was performed using a three-judge LLM panel (adjudication) combined with actual execution results from the Flink cluster.
Datasets
The paper introduces the FlinkEval corpus:
- 3,903 NL–SQL pairs across 45 industry domains.
- 27,321 total questions generated across the 7 levels of the Specification Ladder.
- 404 table DDLs (Data Definition Language).
- Complexity Tiers: Queries are categorized by the number of patterns they combine (Tier 1: single pattern, Tier 3: three or more).
Key Insights and Ideas
- The Information Bottleneck: There is a massive performance gap between explicit requests and vague business questions. While models are good at "translating" technical requirements, they struggle to "infer" missing constraints (like time windows or filters) that a human would consider implicit.
- Streaming is Harder than Snapshot: Models consistently performed 7–11 points worse on streaming queries compared to snapshot (static) queries.
- Pattern Composition is a Major Hurdle: As shown in the complexity tiers, models struggle to maintain accuracy when they have to compose multiple SQL patterns (e.g., a temporal join inside a windowed aggregation).
- Evaluation Sensitivity: The authors found that "face-value" plausibility (does the SQL look like it matches the question?) and "ground-truth agreement" (does it match the gold SQL?) diverge sharply for underspecified requests. A model might generate a perfectly plausible query that is still "wrong" because it missed an unstated business constraint.
Key Results
The evaluation compared frontier models like GPT-6-Astra, Claude Opus 5, and Gemini 3.6 Flash.
- Top Performers: GPT-6 and Claude Opus 5 were the strongest, achieving ~80% pass rates on schema-explicit (L0) requests.
- The Specification Drop: For unclarified requests (L2.1), performance plummeted to ~27%. Adding "oracle clues" (hints about missing details) helped recover some performance, but a significant gap remained.
- Execution Failures: Even for top models, roughly 20% of generated statements failed to execute due to syntax or operational errors on the live cluster.
Limitations and Future Work
- Environment Scope: The benchmark is currently grounded in a single environment (Confluent Cloud). Results might vary on other Flink distributions (e.g., vanilla Apache Flink on-premise).
- Synthetic Ambiguity: The "unclarified" questions are LLM-generated, which might introduce different biases compared to actual human-authored ambiguous questions.
- Execution Accuracy (ExAcc): The authors do not yet report "Execution Accuracy" (comparing the results of the predicted SQL vs. gold SQL) for streaming queries, as this requires complex handling of evolving streams and retractions. This is flagged for future work.
- Interactive Clarification: While the "Ladder" uses oracle clues, it doesn't yet model a multi-turn dialogue where a system asks the user for clarification.
Summary updated