Small open language models fail at turning questions into database queries for two reasons that are usually treated separately. The prompt is cluttered with hundreds of irrelevant table definitions that spread the model's attention thin. The output also drifts into invalid or hallucinated SQL. The standard fix for the second problem is grammar-constrained decoding, meaning a parser that blocks, at every step, any next token that would break the query language. That fix has a known cost. Blocking the tokens the model wanted pushes it onto low-probability paths, and the resulting "reasoning tax" is worst for small models. This proposal argues that the two problems are one problem. The grammar hurts most when the prompt is noisy, because a distracted model puts much of its probability on tokens the grammar must then remove.
The proposed method is a single, training-free decoding pass. The model first writes a short code comment listing the tables it intends to use, with the grammar restricting that comment to real table names. The system then rebuilds the prompt with full definitions only for those tables and their direct foreign-key neighbors. From that point the grammar tracks which tables are in scope and allows only their columns. The model's own commitment drives both the pruning of the prompt and the tightening of the grammar. The main scientific question is whether this pairing lowers the model size at which multi-step, multi-table queries become reliable. That size is the "emergence threshold," below which accuracy is near zero and above which it stabilizes. The comparison is against doing either step alone.
Core Claim
The reasoning tax paid under constrained decoding is not a fixed property of the grammar. It depends on how much of the model's probability the grammar has to throw away. That amount grows with irrelevant context and shrinks as the model's attention concentrates on the right schema elements.
Let $A_t$ be the set of tokens the grammar allows at step $t$, and let $p(v \mid x, y_{<t})$ be the model's next-token probability given prompt $x$ and prefix $y_{<t}$. The retained mass at step $t$ is
$$r_t = \sum_{v \in A_t} p(v \mid x, y_{<t}),$$
and the total distortion of a decoded query is $D = \sum_t -\log r_t$. This is the log-probability the mask removes, which corresponds to the distribution distortion that critics of token-level masking describe.
We predict three things. First, $D$ grows with the number of distractor tables in the prompt and shrinks with model size. Second, the accuracy lost to constrained decoding tracks $D$ across conditions. Third, commit-then-narrow lowers $D$ enough that constrained decoding becomes a net gain for 1.5B–7B models on multi-table queries, where it would otherwise be neutral or harmful. Combined, these would move the scale at which zero-shot Text-to-SQL becomes reliable downward, without fine-tuning or multi-agent pipelines.
Where This Sits
Four lines of work motivate the proposal.
Code-centric schema prompting supplies the input format. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (DAIL-SQL) and Enhancing Text-to-SQL Capabilities of Large Language Models show that CREATE TABLE statements beat prose descriptions of the schema. They also flag the token cost of listing a whole schema this way. We keep the code format but render it only for tables the model has committed to.
Decomposition work supplies evidence that noise from large schemas is what breaks small models. DTS-SQL fine-tunes a 7B model in two stages, first choosing schema items and then writing SQL, and gains 3–7 points. It also reports that roughly 90% schema-linking recall caps accuracy about 5 points below the ceiling. RESDSQL filters columns with a trained cross-encoder before generating a query skeleton. When the filter drops a needed column, the query cannot recover. Accurate Table Question Answering with Accessible LLMs (Orchestra) lifts Mistral-7B from 3.6% to 45.9% on WikiTQ by stripping clutter across three agents. It is limited to single tables and pays for serial agent calls. All three split the task across stages or agents. We keep one decoding pass, and the model does its own schema linking in a form the grammar can check.
SQL-to-Schema Enhances Schema Linking in Text-to-SQL is the closest idea on the pruning side. It drafts a full query, parses out the referenced tables, and regenerates with them highlighted. It reaches 0.981 table-recall@4 with GPT-4-Turbo. It uses two unconstrained generations, keeps the full schema in the prompt, and never tests whether a small model's draft is good enough to prune from. We replace the full draft with a grammar-restricted table header, which is a much easier commitment for a small model.
Grammar-constrained decoding supplies the mechanism, and it is also where the tension lies. Schema-Constrained Grammar-Guided Generation of GQL Queries from Natural Language is the nearest prior work overall. It expands schema placeholders directly into the grammar for on-device models of 7B or less. However, it is a methodology paper without benchmark numbers. It states that context-free grammars cannot enforce which properties belong to which variable. It does not measure the reasoning tax or behavior under noise. BlendSQL constrains generated queries to existing schema entities, but it does not study model scale. Efficient engines such as Earley-Driven Dynamic Pruning for Efficient Structured Decoding and Flexible and Efficient Grammar-Constrained Decoding make per-token masking cheap: 5–32 ms per token for the latter. That is what makes regenerating the grammar in the middle of a decode practical.
The reasoning-tax literature acts as a constraint. It reports that masking distorts the distribution, splits the model's capacity, and cuts off scratchpad reasoning, with the damage disproportionately severe for small models. This could contradict our proposal. If the tax is intrinsic, adding more constraints will make small models worse. Our claim is narrower: the tax is largely a symptom of mass spread over irrelevant context. The retained-mass diagnostic tests this directly rather than assuming it.
Finally, the emergence-threshold literature explains why small models are the regime that matters. Joint success on an $n$-step deduction falls roughly as $P^n$ for per-step accuracy $P$. Work on Turing-machine-style state tracking finds that models under 4B struggle even with the initial steps. Chain-of-thought prompting and self-repair actively hurt below the threshold (Po-Heng Chen; Yixuan Weng et al.). This is why we give small models a short, checkable commitment instead of a free-text scratchpad.
Research Program
The system has three parts.
The first part is the table header. Decoding begins with a SQL comment of the form -- tables: t1, t2. A grammar restricts the comment to table names from the database, which appear in the initial prompt as a compact one-line-per-table list, optionally with column names but no types or keys. We chose a comment, rather than a reordered FROM-first query, because comments are ordinary code the model has seen in pretraining. That should keep the header's own distortion low. A FROM-first variant is tested as an ablation.
The second part is prompt re-rendering. After the header closes, the prompt is rebuilt with full CREATE TABLE definitions for the committed tables and their one-hop foreign-key neighbors. Every other table stays visible only as a name. The key-value cache is recomputed for the new prefix.
The third part is a scope-tracking grammar. The SQL grammar is instantiated so that column tokens in SELECT, WHERE, GROUP BY, and ORDER BY are drawn only from tables bound in the FROM and JOIN clauses, with aliases respected. A JOIN may introduce any table from the name list. This is the escape route for tables missed by the header. When it is used, the new table's definition is spliced into the prompt and its columns join the allowed set. This context-dependent check is exactly what Melo and Pan say a plain context-free grammar cannot do. We implement it as a mask computed from the parser's current bindings, layered on the grammar engine's syntactic mask.
The analysis component instruments every run. It logs $r_t$ per step, split by token type (keywords, table names, column names, literals). It also records header recall and precision against the tables used in the gold query, and how often the JOIN escape is used and whether it helps. These logs separate three failure sources: noise in the prompt, invalid syntax, and invalid table-column pairings.
Baselines
The naive reference point is the same model given the full DAIL-SQL-style CREATE TABLE prompt with free, unconstrained decoding. This follows the representation shown best in Text-to-SQL Empowered by Large Language Models, which reports 68.5% Spider execution accuracy for zero-shot CodeLLaMA-34B and 86.6% for GPT-4 with DAIL-SQL. Every size in our model sweep runs this configuration, which anchors the threshold measurement.
The second baseline adds static schema-aware grammar decoding to the full prompt. The grammar allows only real tables and columns but does no scope tracking. This is our quantitative rendering of the Melo and Pan and BlendSQL approach. Because the reasoning-tax literature says it is most dangerous for small models, it is the configuration in which that tax should be most visible.
The third baseline, which we regard as the strongest training-free competitor, is retriever pruning followed by static grammar decoding. An off-the-shelf embedding retriever keeps the top-$k$ tables, as in the first stage of DTS-SQL but without fine-tuning. Decoding then uses the static grammar. Here $k$ is tuned on a held-out split with the same budget we give our method. This baseline combines "denoise, then constrain" without the model committing to tables itself and without scope tracking. If it matches us, our specific mechanism adds nothing.
The fourth baseline adapts SQL-to-Schema draft-then-refine to the same small models, both unconstrained and with static grammar decoding.
The fifth baseline is sample-and-filter. We draw $N$ unconstrained samples and return the most common result among those that execute. $N$ is chosen so that total generated tokens match our method, including the re-rendering prefill.
The strongest published number on these benchmarks at small scale comes from fine-tuning. DTS-SQL reports 60.31% execution accuracy on BIRD dev and 85.5% on Spider dev with DeepSeek 7B. We report it as a reference, not as a fair comparison, because it uses supervised training. We found no reported result on Lacuna for zero-shot DAIL-SQL prompting of a 7B model on BIRD dev. We will produce that number ourselves.
All methods share the same models, prompts, few-shot examples (none or the same fixed set), grammar engine, and a matched token-and-latency budget.
Two ablations isolate the key ingredients. One ablation keeps the header and re-rendering but removes scope tracking. The other keeps the header and scope tracking but does not re-render the prompt, so the full schema stays in view. Beating a baseline means a higher execution accuracy than the retriever-pruning baseline, on the multi-table subsets, for models of 7B or less, at matched compute. The margin must exceed the spread of a paired bootstrap across three seeds.
Why not just…?
Why not just prune the schema with a retriever and use ordinary grammar decoding? This is the real threat, and the third baseline is built to settle it. Our argument is that an embedding retriever ranks tables by surface similarity to the question. The model's own header is conditioned on how it intends to build the query, and it keeps a recovery path through JOIN, which avoids the unrecoverable false negatives documented by RESDSQL and DTS-SQL. If retriever pruning with static grammar decoding matches our method on BIRD's multi-table questions, we will conclude that the self-commitment and scope tracking are unnecessary and report pruning plus grammar decoding as the finding.
Why not just sample many unconstrained queries and keep the ones that execute? Execution filtering is cheap and avoids any distortion from masking. However, below the threshold, the correct query may rarely appear in any sample. The 3.6% ReAct result for Mistral-7B in Orchestra suggests that more samples do not fix a model that is systematically distracted. Also, BIRD's dirty values mean that queries which execute can still be wrong (Understanding the Effects of Noise in Text-to-SQL). The token-matched fifth baseline decides this empirically, and we will report accuracy as a function of $N$ to show where, if anywhere, the curves cross.
Why not just fine-tune a small model, as DTS-SQL does? Fine-tuning works on seen schema distributions. However, the DAIL-SQL study finds that supervised fine-tuning weakens a model's ability to learn from in-context examples. A fine-tuned checkpoint also cannot follow schema changes without retraining. Our claim concerns zero-shot adaptation. We test it on Spider-DK and Spider-Syn, where train and test schemas and wording differ, and we compare against the fine-tuned numbers as an upper reference rather than claiming to beat them in-domain.
Why not just let the small model reason in free text first, then constrain only the final SQL? This avoids the problem of forcing a commitment before any computation. However, evidence from Po-Heng Chen and Nathan A. Chi et al. indicates that free-text rationales add compounding errors below the threshold. We still include a variant that allows an unconstrained reasoning comment before the header, so the crossover size at which reasoning starts to help can be measured, not assumed.
Proposed Experiments
The primary benchmarks are Spider dev and BIRD dev. We add the corrected-gold subsets of BIRD described in Understanding the Effects of Noise in Text-to-SQL, so that label noise does not obscure the result. Spider-DK, Spider-Syn, and Spider-Realistic test robustness. No numbers are reported on Lacuna for Spider-Syn or Spider-Realistic.
For controlled noise, we inject 0, 10, 50, and 200 distractor tables drawn from unrelated Spider databases into each schema. This keeps the question and gold query fixed while varying only the context.
Models come from one family spanning roughly 0.5B to 32B, so that size is the only variable, with Mistral-7B and Llama-3-8B as cross-family checks. The primary metric is execution accuracy. Secondary metrics are table-recall of the header, the rate of valid table-column pairings, $D$, and wall-clock latency.
Three checks are decisive.
The first check is the interaction. For each model size and distractor count, we compute the accuracy change from adding static grammar decoding to the full prompt. We expect that change to become more negative as distractors grow and as model size shrinks, and the corresponding $D$ to grow with it. If $D$ does not predict the accuracy change across conditions (weak rank correlation), the retained-mass account of the tax is wrong, and the proposal's rationale fails even if accuracy improves.
The second check is the threshold shift. We define the threshold as the smallest model reaching within five points of the 32B unconstrained full-prompt score on BIRD's multi-table questions. We then report how many model sizes down that point moves under commit-then-narrow, compared with the retriever-pruning baseline.
The third check is attribution through the two ablations. This shows whether the gain comes from narrowing the prompt, from narrowing the column mask, or only from their combination. The combination-only outcome is what the core claim predicts. It would also make this the first measurement showing that cleaning the context and constraining the output should be designed together rather than separately.
References
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
- Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
- DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models
- RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL
- Accurate Table Question Answering with Accessible LLMs
- SQL-to-Schema Enhances Schema Linking in Text-to-SQL
- Schema-Constrained Grammar-Guided Generation of GQL Queries from Natural Language
- BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebra
- Earley-Driven Dynamic Pruning for Efficient Structured Decoding
- Flexible and Efficient Grammar-Constrained Decoding
- Computational Reasoning of Large Language Models
- Measuring Taiwanese Mandarin Language Understanding
- Large Language Models are Better Reasoners with Self-Verification
- Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark
- ModeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models