Copy Whole Schema Elements, Not Subword Fragments

Text-to-SQL systems turn a user's question into a database query. On standard benchmarks they look strong, but they collapse when the question stops sharing words with the column and table names. When a question says "pupil" and the column is student_id, accuracy can fall by tens of points. This note tests one specific explanation for part of that collapse.

Modern parsers copy schema names out of their input. They do this through subword tokens, the word fragments that tokenizers trained on general text produce. A name like stu_enroll_dt reaches the model as several unrelated fragments. We conjecture that the model's choice of which column to copy is driven by overlap at the fragment level. One shared fragment with the question can pull attention onto the wrong column. A synonym that shares no fragments gets no pull at all. If this is right, the fix is narrow and cheap: score and copy each schema element as one unit, from a pooled representation that cannot see surface overlap.

The proposal therefore has two parts. The first is a diagnostic that asks whether robustness failures concentrate on fragmented, partially overlapping schema names. The second is an element-level copy head that adds few parameters and plugs into an existing parser. It is evaluated on Dr.Spider, Spider-DK, Spider-Syn and Spider-Realistic. The diagnostic can come out negative. If it does, that is itself a clean result: fragmentation is not the culprit, and the field should look elsewhere.

Core Claim

The claim has two halves. First, among the errors that robustness benchmarks expose, a measurable share comes from copy decisions made at the fragment level. Second, moving the copy decision to whole schema elements, scored without exact-match features, recovers that share. It should do so while staying inside the decoder, so that no separate filtering step can discard the right column before generation starts.

Where This Sits in Prior Work

Three lines of work motivate the claim.

Lexical shortcuts. The robustness benchmarks document the shortcut but stop short of a fix:

  • Dr.Spider applies 17 types of perturbation to questions, databases and SQL. It finds that PICARD loses 14.0% overall and 50.7% on DBcontent-equivalence.
  • Spider-DK adds implicit domain knowledge. It shows RAT-SQL + GAP falling from 67.8% to 44.1%, and names pointing to explicit schema mentions as the failure source. - Re-appraising the Schema Linking argues that exact-match schema linking features act as a benchmark shortcut.

None of these asks at what granularity the pointing goes wrong.

Copying fails under fragmentation. Pointer-Generator Networks for Low-Resource MT shows that a pointer-generator copies "seemingly random subwords" when byte-pair encoding splits cognates into unaligned pieces. A pointer-generator is a decoder that mixes generating from a vocabulary with copying input tokens through attention. That paper studies translation, not structured outputs. We carry its diagnosis to schema identifiers, where fragmentation is extreme because snake_case and abbreviations are common.

The method we reuse. Several papers supply pieces of the method:

  • Get To The Point supplies the soft copy gate.
  • A Copy-Augmented Generative Model for Open-Domain QA shows that a copy head of under 1,600 parameters can ground a large generator. That is the parameter budget we target.
  • LargePiG reads an LLM's own attention as a training-free pointer. It operates at the base tokenizer's fragment level and was never tested on SQL, so it is our natural fragment-level reference.
  • Concept Pointer Network points to higher-level units rather than literal tokens. Its units come from an external concept graph; ours are the schema elements themselves, with no graph needed.

Closest work. Two Text-to-SQL pointer models exist, and neither looks at tokenization:

  • PtrGen-SQL copies question words on single-table WikiSQL.
  • PG-GSQL copies from earlier SQL turns in a dialogue.

One paper complicates the story. RAT-SQL already selects whole columns rather than fragments, yet it still collapses on Spider-DK. So element-level selection alone cannot be the fix. Our version differs in two ways. The element score is computed without exact-match linking features. And during training, the surface strings shown to the copy head are perturbed, so the head cannot learn overlap even implicitly. That ablation is central to the experiments below.

Research Program

Diagnostic. For every schema element and question in Spider dev and its perturbed variants, we compute two quantities using the parser's own tokenizer:

  • Fragment count: how many subword tokens the element's name splits into.
  • Fragment overlap: the fraction of those tokens that also appear in the question.

We then label each wrong column or table choice from fine-tuned T5-style parsers and an open LLM. An error is overlap-attracted if the wrongly chosen element has higher fragment overlap than the gold element. It is overlap-starved if the gold element has zero overlap. Using LargePiG-style attention readouts, we also measure how much copy attention lands on the single most-overlapping fragment of each element.

The hypothesis predicts a specific pattern. Errors under perturbation should rise with fragment count. They should be dominated by the two overlap categories. And they should be predicted better by fragment-level attention than by an element-level similarity computed from pooled hidden states. If error rates are flat across fragment counts, the mechanistic claim fails, and we report it as such.

Intervention: an element-level copy head. At each decoding step $t$, the head scores every schema element $e$ as $$s_t(e) = \mathbf{q}_t^\top W \,\bar{\mathbf{h}}_e .$$ Here $\mathbf{q}_t$ is the decoder state at step $t$. $\bar{\mathbf{h}}_e$ is the mean of the encoder states over all of $e$'s subword tokens, plus its table context. $W$ is a small learned matrix.

A gate $g_t \in [0,1]$ chooses between generating SQL keywords and literals from the vocabulary, and copying an element. If the head copies, it emits all of the element's subwords as one atomic action, so a half-copied identifier is impossible. Exact-match linking features are removed from the input.

During training, the copy head sees schema names that are randomly abbreviated, re-cased or replaced by synonyms. The SQL target stays mapped to the canonical name. This is deterministic string surgery, not LLM-generated paraphrase, and it touches only the schema side. We build the head for two backbones:

  • a fine-tuned T5 parser in the RESDSQL setup, without its pre-filtering cross-encoder;
  • an open LLM, in a training-free variant where pooled attention over each element's fragments replaces LargePiG's per-token pointer.

Baselines

Naive reference. The do-nothing baseline is the same backbone, fine-tuned on Spider, generating schema names fragment by fragment. It uses the same data, steps and seeds as our model. Every gain is reported relative to it, separately on clean and perturbed splits.

Strongest comparable existing method. This is RESDSQL. It ranks whole schema elements with a RoBERTa cross-encoder before decoding, and reports 80.5% exact match and 84.1% execution accuracy on Spider dev and 53.3% exact match on Spider-DK. It is the right competitor for two reasons. It is already element-level, so it directly tests whether doing element scoring inside the decoder adds anything over doing it as a pre-filter. And its known weakness, the error cascade when the filter drops the gold column, is exactly what our design targets. We rerun it at matched backbone size, with the same training data and the same hyperparameter search budget. Beating it means two things. On Spider-DK, our model must improve on its exact match beyond seed variance. On Dr.Spider, our model must lose less relative to clean Spider dev, without lowering clean Spider dev accuracy by more than seed variance.

Pipeline reference. Solid-SQL reports an average 11.6% improvement over baselines on Spider-Syn, Spider-Realistic and Dr.Spider. It uses two-round LLM retrieval and augmentation. We report it as a reference point, alongside its cost, rather than as a matched competitor.

Fragment-level pointer. A LargePiG-style pointer on the same LLM isolates copy granularity. It shares our gate but points to individual fragments.

Key ablation. The decisive ablation keeps the element-level head but restores exact-match features and drops the schema-string perturbation. If this version performs like RAT-SQL on Spider-DK, the gain is shown to come from overlap blindness, not from atomic copying.

Why not just…?

Why not just constrain decoding to valid schema names? PICARD-style constrained decoding guarantees that emitted identifiers exist in the schema. It does not change which valid identifier gets chosen. PICARD itself is the model losing 50.7% on DBcontent-equivalence in Dr.Spider. Even so, this is a real threat if most errors are malformed identifiers. The diagnostic directly measures the split between invalid-name errors and wrong-but-valid errors. If invalid names dominate, constrained decoding suffices and our proposal shrinks to a minor point.

Why not just add schema names to the vocabulary as single tokens? Adaptive Tokenization and Task-Adaptive Tokenization add domain terms as new tokens initialized to the mean of their subwords. They work for classification and generation. But in cross-database parsing the schema changes with every test database, so per-database vocabulary growth would mean new output rows each time. Mean initialization is also weakest on rare, non-compositional identifiers. More importantly, a single-token stu_enroll_dt still carries no meaning connecting it to "pupil." We nevertheless run this as an extra baseline, adding schema tokens at test time with mean initialization. If it matches our head, then the cheaper vocabulary trick wins.

Why not just augment the training data with paraphrases? Improving Generalization by Increasing NL Variation shows that question-side variation helps, and Solid-SQL's augmentation is effective. But Spider-DK reports that models fail to generalize even when the relevant knowledge appears in training. To separate question-side augmentation from our method, we train the naive backbone on the Text2SQL-Flow Spider portion (30,502 examples) with and without our copy head. If the head adds nothing on top of augmentation, we will say so.

Proposed Experiments and Decisive Checks

Benchmarks and metrics. We train on Spider and evaluate on Spider dev, Spider-DK, Spider-Syn, Spider-Realistic and all 17 Dr.Spider categories. We report exact match and execution accuracy, plus a robustness gap defined as clean minus perturbed accuracy. BIRD execution accuracy is a secondary check on messy real schemas. RSL-SQL reports 67.21% there with GPT-4o; we cite that as context, not as a matched comparison.

First decisive check. Does the error-rate slope over fragment count flatten under the element-level head, relative to the naive and fragment-pointer baselines? The diagnostic provides this stratification.

Second decisive check. Do our gains concentrate where the hypothesis says they should?

  • They should be largest on Spider-DK synonyms (T3) and on Dr.Spider schema-synonym and abbreviation perturbations.
  • They should be near zero on implicit-condition and omitted-column cases (T4, T1). No copy mechanism should fix missing domain knowledge there; a gain on those cases would point to an artifact.

Third decisive check. Does the error cascade disappear? We compare RESDSQL with our model on examples where RESDSQL's cross-encoder drops the gold column. Our model has no filter, so it should recover a substantial share of those examples. We also measure added parameters and decoding latency, so the head can be compared with multi-round pipelines on cost as well as accuracy.