FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL

Alfio Massimiliano Gliozzo, Nandana Mihindukulasooriya, Gaetano Rossiello, Erick Lee, Lisa AminiarXiv 2026 · arXiv

TL;DR

FlinkEval is a new, execution-grounded benchmark designed to evaluate how well Large Language Models (LLMs) translate natural language into streaming SQL (Apache Flink). Unlike previous benchmarks that focus on static data, FlinkEval uses a live cluster to validate queries against complex streaming semantics like event time and watermarks, revealing that even frontier models struggle significantly as query complexity increases or user intent becomes underspecified.

What Problem Does This Paper Address?

Most Natural Language to SQL (NL2SQL) research relies on benchmarks like Spider or BIRD, which primarily deal with "bounded" relational data—think of a static Excel sheet or a traditional database. However, modern data engineering often relies on streaming engines like Apache Flink, which process continuous, evolving data streams.

Streaming SQL is a different beast. Success depends on more than just picking the right columns; the query must respect temporal relationships, watermarks, and changelog behaviors. For instance, a query that works on a static table might fail in a stream if it tries to perform a pattern match on data that produces updates rather than append-only events. FlinkEval addresses the lack of evaluation frameworks for these operational and temporal constraints.

Methods

The authors built FlinkEval using an iterative, "execution-grounded" approach:

  1. Synthetic Generation: They created 50 domain schemas and temporally consistent data across 45 industry domains.
  2. Pattern-Driven SQL: They collected 46 SQL patterns from expert-written pairs and vendor documentation, covering complex features like windowing, temporal joins, and MATCH_RECOGNIZE.
  3. Live Validation & Repair Loop: Every generated "gold" query was executed on a live Confluent Cloud Apache Flink cluster. If a query failed or returned no data, it entered a repair loop until it was operationally valid.
  4. The Question Specification Ladder: To simulate real-world ambiguity, they developed a seven-level ladder of natural language requests, ranging from "Schema-Explicit" (technical and precise) to "Unclarified Underspecified" (vague business language).
Table 1: Question Specification Ladder in FlinkEval, ranging from schema-explicit requests to unclarified, potentially underspecified business-language requests; rows run from most to least specified. L2.2–L2.4 augment L2.1 with ranked oracle-derived clues. Results appear in Section 5.2, Table 4.

Evaluation was performed using a three-judge LLM panel (adjudication) combined with actual execution results from the Flink cluster.

Datasets

The paper introduces the FlinkEval corpus:

  • 3,903 NL–SQL pairs across 45 industry domains.
  • 27,321 total questions generated across the 7 levels of the Specification Ladder.
  • 404 table DDLs (Data Definition Language).
  • Complexity Tiers: Queries are categorized by the number of patterns they combine (Tier 1: single pattern, Tier 3: three or more).

Key Insights and Ideas

  • The Information Bottleneck: There is a massive performance gap between explicit requests and vague business questions. While models are good at "translating" technical requirements, they struggle to "infer" missing constraints (like time windows or filters) that a human would consider implicit.
  • Streaming is Harder than Snapshot: Models consistently performed 7–11 points worse on streaming queries compared to snapshot (static) queries.
  • Pattern Composition is a Major Hurdle: As shown in the complexity tiers, models struggle to maintain accuracy when they have to compose multiple SQL patterns (e.g., a temporal join inside a windowed aggregation).
  • Evaluation Sensitivity: The authors found that "face-value" plausibility (does the SQL look like it matches the question?) and "ground-truth agreement" (does it match the gold SQL?) diverge sharply for underspecified requests. A model might generate a perfectly plausible query that is still "wrong" because it missed an unstated business constraint.

Key Results

The evaluation compared frontier models like GPT-6-Astra, Claude Opus 5, and Gemini 3.6 Flash.

  • Top Performers: GPT-6 and Claude Opus 5 were the strongest, achieving ~80% pass rates on schema-explicit (L0) requests.
  • The Specification Drop: For unclarified requests (L2.1), performance plummeted to ~27%. Adding "oracle clues" (hints about missing details) helped recover some performance, but a significant gap remained.
  • Execution Failures: Even for top models, roughly 20% of generated statements failed to execute due to syntax or operational errors on the live cluster.
Table 3: Model comparison on the 666 test-split questions (L0 question, domain DDL context, adjudicated ground- truth-anchored judge panel). Rows 1–2: judge-pass rate (%) and Wilson 95% CI. Execution errors: share of statements that fail to execute (all 81 background statements fail for every model, see text). Pass & ran: share of questions with score 1.0 and no execution error. Slice sizes in parentheses; slices are not weighted to reproduce the corpus composition and snapshot queries exclude the 115 TUMBLE/HOP questions (Section 5.1). Best value per row in bold; pattern groups are in Appendix D (Table 11).

Limitations and Future Work

  • Environment Scope: The benchmark is currently grounded in a single environment (Confluent Cloud). Results might vary on other Flink distributions (e.g., vanilla Apache Flink on-premise).
  • Synthetic Ambiguity: The "unclarified" questions are LLM-generated, which might introduce different biases compared to actual human-authored ambiguous questions.
  • Execution Accuracy (ExAcc): The authors do not yet report "Execution Accuracy" (comparing the results of the predicted SQL vs. gold SQL) for streaming queries, as this requires complex handling of evolving streams and retractions. This is flagged for future work.
  • Interactive Clarification: While the "Ladder" uses oracle clues, it doesn't yet model a multi-turn dialogue where a system asks the user for clarification.

Summary updated

Abstract

Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question--SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.
BibTeX
@misc{gliozzo2026flinkeval,
  author = {Alfio Massimiliano Gliozzo and Nandana Mihindukulasooriya and Gaetano Rossiello and Erick Lee and Lisa Amini},
  title = {{FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL}},
  year = {2026},
  eprint = {2611.17093},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17093},
}
Figure