When a user talks to a database over several turns, each new question usually changes the previous query rather than starting over. Sometimes the change is a retraction: "actually, forget the 2019 filter." Current systems, from fine-tuned parsers to frontier language models, regenerate the whole SQL query at every turn from the dialogue history. On the newest benchmark for this setting, models almost always notice that the user changed their mind. They then often fail to act on it. In Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift, drift is recognized more than 94% of the time but resolved correctly only about 45–55% of the time. The gap is worst when a prior WHERE condition must be discarded. Our hypothesis is that this is a decoding problem more than an understanding problem. Free regeneration lets an old condition be copied forward silently, because nothing forces the model to decide what happens to it.
We propose to keep an explicit, numbered ledger of the clauses in the current query: each projection, condition, value, grouping and ordering is one entry. At every turn the model emits an edit script instead of a query. A grammar enforced during decoding requires the script to state exactly one decision for every existing entry, in order: keep it, drop it, or replace it. Only after that can new clauses be added. Grammar-constrained decoding (masking any next token that would break a formal grammar) normally guarantees well-formed output. Here we use it for a different purpose: to make a silent carry-over of a stale commitment impossible to express. New subtrees remain subject to SQL syntax, schema checks and simple type checks, so the compiled query is always valid.
Core Claim
The claim is that forcing the decoder to account for each prior commitment will close most of the recognition-resolution gap on constraint-level drift, at no cost on monotonic dialogues. The key ingredient is coverage: the grammar admits only scripts that touch every ledger entry. An edit format alone is not enough.
The ledger also serves as a joint syntactic-semantic representation of the dialogue. Each entry is a SQL subtree, the syntax, paired with its semantic role (value, constraint or projection) and the column it binds. User phrases such as "that filter" or "instead of revenue" can then be resolved against something concrete. This follows the CCG tradition of binding semantic roles directly onto syntactic argument slots, as in Joint A* CCG Parsing and Semantic Role Labelling. It also follows the finding in Joint Universal Syntactic and Semantic Parsing that passing an explicit intermediate structure to a semantic decoder helps it. We apply that idea across turns rather than within a sentence. We treat global coherence as the evaluation lens: a multi-turn dialogue is a short narrative whose state must stay consistent, and drift errors are contradictions of the user's latest premise.
Relation to Prior Work
The closest systems split along the two halves of our design. PICARD constrains each SQL query with incremental parsing and schema guards during beam search. It has no notion of state carried across turns, so it validates whatever query the model regenerates, stale conditions included. CQR-SQL trains contextual and rewritten standalone questions into a shared latent space through consistency losses, then decodes without constraints. Both of its objectives assume that context accumulates monotonically.
Conversational Text-to-SQL: An Odyssey adds rerankers on top of PICARD on CoSQL and finds a compositional collapse: above 60% exact match on query structures seen in training, 12.5% on unseen ones. An edit-based output should help here, because it reuses existing subtrees instead of recomposing whole trees.
Two papers supply our machinery. GRAMMAR-LLM compiles prefix-safe grammars to deterministic pushdown automata, which gives us the decoding engine. It also states the limitation we target: valid output can still be semantically wrong. ChopChop shows that semantic checks inside the decoder, such as type safety, are feasible. It also warns of 50–200 ms of overhead per token, so our type checks are limited to cheap value–column compatibility tests. None of these works makes "what happens to each earlier commitment" an obligatory part of the output grammar.
Research Program
We first build the ledger and the edit grammar. Gold edit scripts for training come from the abstract syntax trees of consecutive gold queries in SParC and CoSQL: we align subtrees between turn $t-1$ and turn $t$ and label each prior entry as kept, dropped or replaced. Remaining new subtrees become additions.
At decoding time, the model sees the dialogue history and a rendering of the current ledger, with entry IDs and their semantic roles. It emits the script under three nested constraints:
- A pushdown automaton that enforces exactly one ordered decision per entry ID.
- PICARD-style SQL grammar and schema guards on any new or replacement subtree.
- A type guard that rejects literal values incompatible with the column's type.
The script is compiled deterministically into SQL.
We run the method in two regimes. The first fine-tunes an open model on SParC and CoSQL edit scripts. The second applies it zero-shot to open instruction-tuned models on TIDE-BENCH, using only a format description and grammar masking. This shows whether coverage helps without task-specific training.
Two analyses accompany the main results. We introduce a silent-carry-over rate: the fraction of turns where the output still contains a condition that the gold query dropped. We also report accuracy by turn depth, to see whether the steep decline across turns flattens.
Baselines
Naive reference. An unconstrained open model, the same one used by our method, generates full SQL from the concatenated history. This anchors how much any structure adds.
Strongest existing method on TIDE-BENCH. This is the best frontier model reported there. Claude-Sonnet-4.6 reaches 27.8% Dialogue Execution Accuracy on the Joint chain-plus-drift samples, with drift resolution accuracy around 45–55% across models, per Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift. We cannot match its compute. We therefore compare against the published numbers and, where an API allows, re-run it with our exact prompts. Beating it with an open model of 7–13B parameters on constraint-level drift resolution would be a strong result. Matching it would still be informative.
Strongest existing methods on SParC and CoSQL.
- CQR-SQL reports 68.0% question match and 48.8% interaction match on the SParC development set, and 58.5% and 31.1% on the CoSQL development set (CQR-SQL).
- PICARD reports 75.5% exact-set-match on the Spider development set with T5-3B. We found no reported result on Lacuna for PICARD's CoSQL numbers in the material available.
We re-implement PICARD-constrained full-query regeneration on our backbone, with the same training data and the same beam size of 4. This gives a fair internal comparison.
Key ablations. Two ablations isolate our ingredient:
- Edit scripts decoded under SQL constraints but without the coverage requirement, so the model may omit entries and omitted entries default to keep.
- Full-query regeneration with the previous gold-or-predicted SQL shown in context.
All systems get the same tuning budget, measured in hyperparameter trials. Beating the ablations means a statistically significant gain in drift resolution accuracy on constraint-level drift, and a lower silent-carry-over rate. On monotonic SParC turns, question match must be no worse than the regeneration baseline within noise.
Why not just…?
Why not just use a frontier model with the full history? Scale is already measured here. The best of twelve models reaches 27.8% Dialogue Execution Accuracy on the Joint samples, and the gap of more than 40 points between recognizing and resolving drift persists across models (TIDE-BENCH paper). The Odyssey authors also report that scaling does not fix the 12.5% unseen-structure collapse. Larger models may still close part of the gap, so we report our zero-shot ledger decoding on the largest open model we can run.
Why not just show the previous SQL and ask the model to edit it? This is the real threat. Much of the gain could come from having the old query visible rather than from coverage. Our second ablation is exactly this baseline. If it matches the ledger with coverage on constraint-level drift, our hypothesis is wrong: the problem is missing state, not unforced accounting. We would report that outcome.
Why not just rewrite each turn into a standalone question and parse it? Rewriting pipelines pass rewriter errors downstream. CQR-SQL moved rewriting into training precisely because its latent version outperformed explicit rewrite-then-parse. A rewriter also has no mechanism that obliges it to drop a retracted condition. We include CQR-SQL's published numbers. Where TIDE-BENCH databases allow, we also run a rewrite-then-PICARD pipeline on our backbone.
Why not just sample many queries and filter? Filtering can reject invalid SQL but cannot detect a valid query that kept a stale condition. Execution-based filtering cannot either, because both queries run. PICARD also shows that post-hoc filtering needs beams of 16–100 to match what masking achieves with a beam of 4.
Decisive Checks
The central test is drift resolution accuracy on TIDE-BENCH's constraint-level drift subset, comparing the ledger with coverage against the same ledger without coverage, on an identical backbone. If coverage does not produce a clear gain there, together with a drop in silent carry-over, the core claim fails.
Secondary checks have four parts:
- Interaction match and accuracy by turn depth on SParC and CoSQL, where we expect a flatter decline beyond turn 3.
- Exact match on the unseen-structure split defined by the Odyssey analysis.
- Per-token latency against PICARD's reported 3.1 s versus 2.5 s per sample.
- Chain-ambiguity identification, where Layer-2 detection is often below 15%. We report it as a non-target control; we expect little change there, since the ledger addresses retraction, not clarification.
References
- Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
- Joint A* CCG Parsing and Semantic Role Labelling
- Joint Universal Syntactic and Semantic Parsing
- PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
- CQR-SQL: Conversational Question Reformulation Enhanced Context-Dependent Text-to-SQL Parsers
- Conversational Text-to-SQL: An Odyssey into State-of-the-Art and Challenges Ahead
- GRAMMAR-LLM: Grammar-Constrained Natural Language Generation
- ChopChop: A Programmable Framework for Semantically Constraining the Output of Language Models