Large language models score above 80% on Spider, the standard benchmark for translating English questions into database queries. Part of that score is recall rather than reasoning. When the same kind of questions are posed over a hidden, never-published database, accuracy drops sharply. On the Termite benchmark, GPT-3.5's accuracy on hard queries falls from 48.83% to 28.33% (Ranaldi et al.). At the same time, the strongest systems split the task into stages. One stage decides which tables and columns matter. Another writes a query skeleton. A third breaks nested questions into sub-questions. These staged systems are credited with better generalization. Nobody knows which stage the memorized benchmark knowledge flows through, or whether staging reduces reliance on memory at all. Staging might just split one contaminated lookup into two.
This note proposes to answer that question with one instrument, which then becomes a method. Take a staged parser and replace every schema name and cell value the planning stage would see with a typed placeholder: a neutral token that keeps the kind of thing it stands for, such as [COL_2:number] or [VAL_1:date], but drops the name. Then measure how much of the accuracy gap between memorized and unseen databases disappears. If the planner is where memory is exploited, masking it should shrink the gap while costing little on unseen data. That would argue for a parser that plans over placeholders and restores real names only at the last step.
Core Claim
The claim has two parts that can be tested separately.
The first part is diagnostic. In a decoupled pipeline, memorized benchmark knowledge enters mainly through the planning stages, not through schema linking. Planning here means the skeleton generator or question decomposer. Schema linking is the stage that matches question words to columns. Its errors are real but mostly not memory-driven.
The second part is constructive. Suppose placeholders are carried consistently through a tree of sub-questions, so that sub-questions inherit, pass along and finally ground the same placeholders. Then the planner keeps its accuracy on genuinely unseen databases while losing only the part of its Spider score that came from recall.
The quantity of interest is the contamination gap for a system $s$, evaluated at matched query difficulty $h$:
$$G_s(h) = \mathrm{EX}_s^{\text{Spider}}(h) - \mathrm{EX}_s^{\text{Termite}}(h)$$
Here $\mathrm{EX}$ is execution accuracy, meaning the fraction of predicted queries whose results match the gold query's results. The hypothesis is that masking the planner shrinks $G_s$ substantially. It should also leave $\mathrm{EX}^{\text{Termite}}$ unchanged or higher. A drop on Spider alone would not count as success.
Relation to Prior Work
Three papers already combine two of these ingredients, and each stops short of this question.
Data-Anonymous Encoding replaces columns and cell values with abstract tags before parsing. It learns the tagging with policy gradients, and gains +6.4% over Seq2SQL on WikiSQL and up to 3.2% on Spider. It supplies our masking mechanism. However, it works on flat, single-step queries and never asks whether anonymization affects memorization.
DIN-SQL and Complex Question Decomposition for Semantic Parsing (HSP) supply the hierarchical decomposition. DIN-SQL breaks nested questions into sub-problems. HSP generates sub-questions and copies entities into them. Both keep raw entity strings in every step, which is exactly the channel through which memorized associations could act.
Semantic Decomposition of Question and SQL contributes QPL, a tree of relational operations such as scan, filter, join and aggregate. We use it as the plan format, with placeholders in place of schema names.
Ranaldi et al. motivate the work. They showed that masking 25% of column names lets GPT-3.5 reconstruct Spider schemas at 33.42% versus 13.21% on Termite. They also showed that deleting foreign keys barely hurts on Spider but collapses performance on Termite. They use masking only as a static probe of one-pass prompting. We move masking inside a staged pipeline, stage by stage, as both a measurement and a design choice.
RESDSQL is the paper whose generalization claim we test. It reports 80.5% exact match and 84.1% execution accuracy on Spider dev by separating schema ranking from skeleton decoding.
DyVal constrains our evaluation design. It shows that regenerating problems with permuted surface order exposes memorization. That motivates our regenerated-renaming split, although DyVal itself covers only synthetic arithmetic and logic.
Research Program
We build a four-stage parser.
The first stage is a schema linker, a cross-encoder ranker in the RESDSQL style. It proposes relevant tables, columns and matched values.
The second stage is a masker. It rewrites the question and the retained schema into typed placeholders. Types are drawn from column data types plus value categories such as number, date, text and identifier. Keeping types addresses the known cost of delexicalization, namely losing cues about whether a number is an age or a year. Foreign-key edges between placeholders are kept as structure.
The third stage is a hierarchical planner. It decomposes the masked question into a tree of sub-questions and emits a QPL plan whose leaves refer only to placeholders. Sub-questions inherit placeholder bindings from their parents, so [TABLE_1] means the same table throughout the tree.
The fourth stage is a grounder. It substitutes real identifiers back and compiles the plan to SQL common table expressions, which are named sub-queries.
The same pipeline runs in two ways. The first is a fine-tuned T5-3B, trained on masked Spider plans converted from Spider-QPL. The second is an in-context LLM variant using DIN-SQL-style prompts over masked inputs.
The diagnostic study turns masking on or off at each stage independently. It also swaps in oracle outputs for each stage, meaning gold links, gold plans or gold grounding. This attributes the contamination gap to stages: if giving the planner real names restores Spider accuracy but not Termite accuracy, the planner is where recall happens. To control for memorization that masking cannot remove, each model's DC-accuracy (Ranaldi et al.) is measured and reported alongside the gap. A model that cannot reconstruct Spider schemas should show little gap to remove.
Baselines
The naive reference point is single-pass prompting of the same LLM with the full schema and question. This is the setup Ranaldi et al. studied. They report 90.11% (easy) and 48.83% (hard) on Spider versus 74.00% and 28.33% on Termite for GPT-3.5. We rerun it with our chosen models, because those numbers are for one older model.
The strongest existing staged LLM method is DIN-SQL. It reports 85.3% execution accuracy on the Spider test set and 55.9% on BIRD with GPT-4 (DIN-SQL). No reported result was found on Lacuna for DIN-SQL on Termite. Its gap is therefore something we must measure ourselves, and it is the comparison that matters most.
The strongest fine-tuned staged methods are RESDSQL-3B with NatSQL, at 84.1% execution accuracy on Spider dev, and Text-to-QPL on Flan-T5-XL, at 83.8% on Spider dev (Eyal et al.). Both are retrained on our data splits with the same backbone size and the same number of tuning runs as our fine-tuned variant. All LLM variants use the same model, the same few-shot examples where applicable, and a matched budget of calls per query.
The key ablation is our own pipeline with the masker disabled. It is the same linker, planner and grounder with real names passed through, so any gap reduction can be traced to masking alone. A second ablation masks only the root question, as DAE does, and not the sub-questions. This isolates the value of carrying placeholders through the tree.
Beating the baselines means two things at once. Our pipeline must match or exceed DIN-SQL and RESDSQL on Termite execution accuracy, within confidence intervals computed by resampling questions. It must also show a smaller contamination gap $G_s$ at easy, medium and hard difficulty.
Why not just…?
Why not just rename the test schemas and feed them to DIN-SQL? Consistent renaming of identifiers is the main threat, because it is the simplest way to deny recall. The weakness is that it destroys meaningful column names, which a legitimate parser needs. Accuracy losses under renaming then mix lost memory with lost meaning. Our design masks only what the planner sees; the linker and grounder still read real names. The experiment settles this directly. We run DIN-SQL on a regenerated Spider split whose schemas are renamed to synonymous but novel identifiers, following DyVal's idea of regenerating surface form. We compare that with our pipeline on the original split. If renaming alone closes the gap as much as masking the planner does, at equal Termite accuracy, the constructive claim fails and the contribution reduces to the diagnostic study.
Why not just evaluate on Termite and stop? Termite shows that a gap exists but not where it comes from. It is also only one static set, which will eventually leak. The stage-wise attribution is the part Termite cannot provide. The regenerated split gives a renewable second check.
Why not just use a stronger model in one pass? Frontier models still fail on hard compositional settings: o1-preview solves only 21.3% of Spider 2.0 tasks (Spider 2.0). More importantly, a stronger model has likely memorized more of Spider, not less. We include the one-pass baseline at the largest model we can afford and report its gap. If the gap is small and Termite accuracy is high, masking is unnecessary for that model, and we will say so.
Why not use string matching for the placeholders instead of a learned masker? DAE found that learned anonymization beats deterministic matching. Dr.Spider shows parsers dropping 50.7% under database-content perturbations, which surface matching invites. We still include a string-match masker as a cheap variant. If it performs as well, we keep it, since the claim concerns what the planner sees, not how the masks are made.
Proposed Experiments
The primary evaluation compares Spider dev with Termite at matched difficulty. It reports execution accuracy and the gap $G_s$ for every system and every per-stage masking configuration, together with oracle-swap results that attribute the gap to stages. The secondary evaluations cover three settings. The first is Dr.Spider's 17 perturbation types, with attention to the database-content and synonym categories. The second is the regenerated renamed split. The third is BIRD, where masking of dirty cell values is tested against GPT-4's reported 54.89% with external evidence (BIRD).
The decisive check is a single comparison. With the planner masked, does Termite hard-query accuracy stay at or above the unmasked pipeline's, while the Spider–Termite gap shrinks by a clear margin beyond resampling noise? Two outcomes would falsify the constructive claim. One is that masking lowers Termite accuracy; that would mean typed placeholders discard information the planner genuinely needs. The other is that the gap shrinks only because Spider accuracy falls. In either case the result reverts to a diagnostic finding about where memorization lives.
References
- Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL translation
- Data-Anonymous Encoding for Text-to-SQL Generation
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Complex Question Decomposition for Semantic Parsing
- Semantic Decomposition of Question and SQL for Text-to-SQL Parsing
- RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
- Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs