Many database questions are ambiguous only because of a conjunction. "Show customers in Ohio with overdue invoices and open tickets" can mean customers who have both, or customers in Ohio with overdue invoices plus anyone with open tickets. "Organic apples and oranges" may or may not put "organic" on the oranges. Language models turning such questions into SQL usually pick one reading without saying so, and the user never learns that another reading existed. We propose that when a question has several valid scope readings, the model should return a single machine-parseable object that lists each reading as its own executable query, together with the bracketing it assumes. When the question has only one reading, the object should contain exactly one. We also propose to measure this on freshly generated databases and questions, so that high scores cannot come from memorized benchmark items.
This takes the idea of "Overton pluralism", meaning one response that covers the full range of reasonable views instead of a range spread across repeated samples, and applies it to a syntactic problem where the reasonable views can be enumerated and checked by execution. The contribution has two parts that support each other. The first is a procedural generator of coordination-ambiguous text-to-SQL items with known readings. The second is a decoding method that enumerates bracketings under a grammar and forces each branch of the output to commit to a different one. The generator is what makes the method's claims testable.
Core Claim
Existing work on ambiguous text-to-SQL treats ambiguity as a diversity problem over separate outputs. Benchmarking and Improving Text-to-SQL Generation under Ambiguity introduces AmbiQT, where each question has two valid SQL readings. Its LogicalBeam method perturbs abstract query plans and fills them in under constraints, so that a top-k list covers both readings. AmbiQT's four ambiguity types are column, table, join and precomputed aggregate, so coordination scope is not among them. Its outputs are also a ranked list handed to a human, and the benchmark is static.
S²SQL tackles coordination in text-to-SQL by injecting dependency edges into the encoder. It fixes omitted conjoined columns, but it commits to one parse and does not treat scope as ambiguous.
Grammar-constrained decoding guarantees parseable output. It prunes illegal tokens at each step, as in Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning and SynCode. Both papers are explicit that this guarantees syntax, not meaning.
Our claim is that scope ambiguity is small and discrete: a coordination with a pre- or post-modifier has a handful of possible bracketings. The set of readings can therefore be enumerated from the input. It can then be imposed as an input-dependent grammar on a structured output, in the spirit of the input-dependent grammars that lifted entity disambiguation accuracy from 54.1% to 80.3% in the GCD paper. A distinctness constraint can then stop the model from writing the same reading twice. We predict this beats both unconstrained "list all readings" prompting and LogicalBeam-style plan perturbation on scope coverage. We further predict that the advantage holds on freshly generated items where memorization cannot help.
Research Program
The generator. The first artifact extends the logic of DyVal to a linguistic task. DyVal builds reasoning problems from randomly generated graphs so that every test item is new, and it explicitly lists non-formal tasks as outside its reach. Our generator samples a small relational schema, a coordination template, and a modifier position. The coordination can be flat or nested, as in "(A and B) or C". The modifier can sit before the coordination, after it, or as a relative clause. From the resulting bracketings the generator derives the gold SQL for each reading.
It then fills the database with rows chosen so that every pair of readings returns different results. This makes execution a reliable referee between readings.
Each item carries a disambiguation level, and there are three:
- Structural. Nothing in the schema or world knowledge rules out any reading.
- Schema-disambiguated. The modifier's column exists in only one table, so one reading is invalid.
- Unambiguous control. The question has exactly one reading.
The unambiguous controls matter. They are the analogue of the Overton window's boundary problem: a system that hedges on everything should be penalized, just as a pluralistic answer that includes fringe views would be.
Column and table names are drawn from two lexicons: high-frequency English nouns and plausible but rare compounds. Following the frequency analysis in Snoopy, which found few-shot accuracy tracks pretraining term frequency, this lets us test whether scope resolution leans on familiar surface strings.
The decoder. The second artifact is a two-stage constrained decoder. In the first stage, a lightweight coordination detector proposes coordinators and candidate conjunct boundaries. We reuse the two-stage design of CoRec, a BERT coordinator classifier followed by a boundary tagger. From these candidates we compile an input-dependent grammar for a JSON object whose top level is a list of readings. Each reading contains a bracketing string, restricted to the bracketings the detected structure allows, and a SQL string, restricted by a SQL grammar and the schema.
The decoder carries one extra piece of state: the set of bracketings already emitted. When the model opens a new reading, bracketings already used are masked out. The model is also allowed to close the list after one reading, and on unambiguous items this is the correct behavior.
SQL infilling runs under the same grammar and mask machinery SynCode uses. That keeps structural validity at the 100% level SynCode reports on JSON, while the plan level carries the semantic diversity. This split between plan and infill mirrors LogicalBeam. The differences are that the plans here are coordination bracketings derived from the input, and that all readings sit in one validated object rather than a ranked list.
Metrics. Let $G$ be the set of gold readings for an item and $E$ the set of readings the system emits. We count an emitted query as matching a gold reading when their execution results on the generated database are identical. Scope coverage is the fraction of $G$ matched by some query in $E$. Scope precision is the fraction of $E$ that matches some reading in $G$. Their harmonic mean is our headline number. It borrows the set-coverage view of pluralism from Benchmarking Overton Pluralism in LLMs, but replaces an LLM judge with execution, which is possible only because our readings are enumerable.
We also report three further measures. The first is over-hedging rate, the fraction of unambiguous controls with more than one reading. The second is structural validity. The third is the contamination gap, the difference in scores between AmbiQT/Spider-derived items and generated items that are isomorphic to them.
Baselines
The naive reference point is greedy single-query generation from the same open-weight model under a SQL grammar. It gets full structural validity but, by design, covers at most one reading. Its coverage on structural items shows how often models silently drop a reading.
The strongest simple competitor is prompted enumeration under schema-constrained decoding. The same model receives few-shot examples of ambiguous questions mapped to lists of readings. It decodes under a JSON schema that allows a variable-length list of SQL strings, enforced with SynCode, so formatting failures cannot explain any gap. It gets the same examples and the same prompt-tuning budget as our method. It lacks only the input-derived bracketing grammar and the distinctness mask. For closed-API models we run the prompt-only version and report it separately, since they cannot be masked.
The strongest existing method is LogicalBeam. We reimplement it on the same base model with top-k equal to the maximum number of readings in the generator, and score its list with the same coverage and precision metrics. On its own benchmark it is reported to be 1.5x to 2.5x more effective than baselines at getting both readings into the top-5 on AmbiQT. It also raised top-5 accuracy on Kaggle DBQA from 26.5% to 35.4% (Benchmarking and Improving Text-to-SQL Generation under Ambiguity). No reported result found on Lacuna for LogicalBeam or any system on coordination-scope ambiguity specifically, so the comparison on our generator is new. To count as beating LogicalBeam, our method must achieve higher scope coverage at equal or better scope precision and a lower over-hedging rate, across seeds and at all three disambiguation levels.
We also include a supervised baseline: the same base model fine-tuned on the generator's training templates. It is evaluated on held-out templates and lexicons, with a compute budget matched to our detector's training.
Three ablations isolate the key ingredient:
- Remove the distinctness mask but keep the bracketing grammar.
- Replace the detector with brute-force enumeration of all bracketings, which tests whether the detector's pruning matters.
- Supply gold bracketings, an oracle that gives the ceiling on what better detection could deliver.
Why not just…?
Why not just prompt the model to list all readings in JSON? This is the real threat. With SynCode enforcing the schema, formatting failures disappear, so the ~64% structural-validity rate of unconstrained decoding in the GCD paper no longer argues against it. What remains open is whether models notice scope ambiguity unprompted, and whether they avoid writing near-duplicate readings. The "distinct plans" comparison against the prompted-enumeration baseline settles this directly. If prompted enumeration matches our coverage and precision, we will report that the decoder adds nothing. The contribution then narrows to the benchmark, the over-hedging analysis and the contamination gap, which remain useful on their own.
Why not just sample or beam-search several queries? AmbiQT found that standard beam and top-k decoding produce token-level variation such as renamed table aliases rather than different logical readings. From Distributional to Overton Pluralism found that extra distributional samples tend to be lower-quality than a synthesized single response. We include temperature sampling with the same number of candidates as an inexpensive check. We expect it to trail LogicalBeam, and we do not treat it as a serious contender.
Why not just fine-tune on ambiguous queries with multi-reading targets? Because the generator produces unlimited training data, this is fairer here than usual, and we run it. The risk is template memorization. Spider reports TypeSQL falling from 34.3% to 9.7% on unseen databases, and the GCD paper shows an unfinetuned constrained LLaMA-33B beating a supervised GenIE model on closed extraction (F1 36.0 vs 34.8). The held-out-template split will show whether fine-tuning generalizes across coordination shapes it did not see.
Why not just run a syntactic parser first? A parser returns one tree. That is exactly the collapse to a single reading we want to avoid. Full parsers are also slow: AllenNLP took 823 seconds versus CoRec's 57 on Penn (Simple). We use a CoRec-style detector only to propose candidate boundaries, never to choose among readings.
Decisive Checks
We will evaluate open-weight models with logit access, following the GCD paper's LLaMA setting, on three sources:
- generated items at all three disambiguation levels;
- AmbiQT and Spider dev, to establish that the method does not hurt standard accuracy;
- generated items isomorphic to Spider questions, to measure the contamination gap.
The idea is confirmed only if three things hold at once. The bracketing-constrained decoder must beat both prompted enumeration and LogicalBeam on scope coverage at matched precision on fresh items. Its over-hedging rate on unambiguous controls must stay low. Its advantage must not shrink when rare lexicons replace common ones, whereas we expect the baselines' scores to drop more on rare names if they lean on familiar strings.
A large gap between static and generated items for every system would be a finding in its own right. It would show that current text-to-SQL scores overstate how well models handle conjunction scope. If the gold-bracketing oracle far exceeds the full system, detection is the bottleneck, and the next step is better boundary detection rather than better decoding.
References
- Benchmarking and Improving Text-to-SQL Generation under Ambiguity
- S²SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- SynCode: LLM Generation with Grammar Augmentation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- CoRec: An Easy Approach for Coordination Recognition
- Benchmarking Overton Pluralism in LLMs
- From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
- Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task