Alignment Fragments as In-Context Curricula for Semantic Parsing

Large language models can turn questions into programs, such as SQL queries or calendar commands, when they are shown a few worked examples in the prompt. This ability is fragile. Rewording the instruction, reordering the examples, or changing their formatting can swing accuracy by double digits. An older literature on low-resource alignment offers a lever for this. Alignment here means marking which words of the input produce which piece of the output. That work found that a few dozen hand-made alignments beat millions of unannotated pairs, because they tell the model exactly which phrase maps to which piece of structure. So far, though, alignments have only been used to train model weights. We propose using them inside the prompt instead. Each of a handful of hand-aligned examples is cut into its aligned pieces ("number of medals" → count(medal); "won gold" → medal = 'gold'). The pieces are then presented as a curriculum: atomic phrase-to-fragment mappings first, then the sub-programs built from them, then one or two complete examples.

The claim is that this "fragment curriculum" does two things whole-example prompts do not. It gives the model a grounded meaning for each phrase, which should make its output depend less on how the instruction happens to be worded. It also gives the curriculum a natural difficulty order that needs no external metadata or trained analyzer. That order is subtree depth, and later fragments are literally built from earlier ones. The second property also yields a testable explanation for a contradiction in the curriculum literature about whether demonstration order matters.

Core Claim

With only 8 to 64 hand-aligned training examples, a prompt built from aligned sub-program fragments should do better than whole-example prompts of equal token length, ordered from shallow to deep with each fragment's parts appearing before it. We expect it to improve exact-match or execution accuracy on compositional semantic parsing. More distinctively, we expect it to reduce the share of performance variance caused by semantics-preserving prompt changes.

We further hypothesize that demonstration order matters causally only when demonstrations are compositionally nested, meaning later ones reuse earlier ones. When they are merely difficulty-stratified, order should not matter. This would reconcile Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning, which found that shuffling its easy-to-hard examples hurt, with Curriculum Demonstration Selection for In-Context Learning, which found that shuffling did not. The first method retrieves examples whose reasoning operations are subsequences of the query's. The second samples one independent example per difficulty partition.

Where This Sits

Three lines of prior work motivate the proposal by showing that small amounts of explicit alignment are unusually valuable.

All three consume alignments through gradient updates. We consume them through the prompt, with no weight updates. The third paper also derived its spans from heuristic pre-aligners, whereas we test whether a few gold alignments matter more than automatic ones.

The in-context curriculum methods supply the ordering idea but use weaker difficulty signals:

  • Chain-of-Quizzes orders model-verified examples by text length.
  • Curriculum Demonstration Selection uses external difficulty tags.
  • The problem-solving-logic curriculum (PSL-ICL) relies on a decomposition model fine-tuned on the full BREAK dataset. That model maps questions to a fixed set of 13 reasoning operations.

None of them ties reasoning steps back to specific input spans. Our curriculum is defined by the alignment itself, so its difficulty signal is grounded in the input text, needs no extra annotation, and is compositionally nested by construction.

The prompt-sensitivity literature provides the evaluation lens and one constraint.

  • Brittlebench splits variance into a part due to item difficulty and a part due to prompt perturbation. Prompt perturbation accounts for up to about 50% of variance in open-weight models. Few-shot prompting increased formatting brittleness, and chain-of-thought (asking the model to write out intermediate reasoning) shrank the perturbation-induced drop by only about 17%.
  • The Order Effect found that few-shot prompts made shuffling damage worse.
  • ProSA found the opposite pull: one demonstration "anchors" the model and lowers sensitivity, and low decoding confidence goes with high sensitivity.
  • On the Relation between Sensitivity and Accuracy in In-Context Learning showed that sensitivity correlates negatively with correctness (Pearson correlation as strong as −0.40).

Our proposal takes a side in this tension. It predicts that what makes demonstrations stabilize the model or destabilize it is whether they ground phrases in structure, not how many there are.

Finally, Flaw or Artifact? constrains the design. It showed that rigid string-match scoring inflates apparent sensitivity: Gemma's standard deviation on ARC-Challenge fell from 0.28 to 0.005 under an LLM judge. We therefore score SQL by execution and score programs by canonicalized tree equality. Formatting differences then cannot masquerade as brittleness, or as a curriculum gain.

Research Program

Annotation. For each benchmark we fix a small seed pool of $k \in \{8, 16, 32, 64\}$ training examples, each with a gold span-to-subtree alignment. For SQUALL these alignments already exist. For SMCALFLOW-CS and the CFQ Recursive split, we hand-align the seed pool ourselves and log the annotation time. The prior heuristic alignments for those datasets serve as the automatic-alignment condition.

Fragment extraction. Each aligned example is decomposed into fragment demonstrations. Every aligned subtree becomes a pair consisting of its input span and its sub-program, rendered in the same prompt format as a full example. Each fragment is labeled with its depth in the program tree.

Fragment selection. For a test query we pick fragments in three steps:

  1. Pick fragments whose spans overlap lexically with the query.
  2. Close this set under "parts-of", adding the children of any selected fragment.
  3. Fill the remaining token budget with one or two full seed examples whose programs share the most fragments with the selected set.

Ordering. The prompt is ordered by depth, with every fragment's children appearing before it, and ends with the full examples. Selection and ordering are deterministic and cheap. They need no trained analyzer and no quiz-style model calls. This keeps the method within the low-resource regime it targets.

Perturbations. We measure robustness with a fixed perturbation set drawn from the sensitivity literature:

  • ten instruction paraphrases;
  • formatting changes (separators, whitespace, field labels);
  • permutations of demonstration order.

Sensitivity metric. Following Brittlebench, the headline sensitivity number is the brittleness share

$$S = \frac{V_{\text{prompt}}}{V_{\text{prompt}} + V_{\text{data}}},$$

where $V_{\text{prompt}}$ is the variance of per-item correctness across perturbations and $V_{\text{data}}$ is the variance across items at a fixed prompt. We also report ProSA's PromptSensiScore and the mean log-probability of the gold program. The latter tests ProSA's proposed mechanism: fragments should raise decoding confidence on the parts of the query they cover.

Models. We use open-weight models, where Brittlebench found sensitivity largest. These are Llama3-8B, to match PSL-ICL, plus one model in the 70B class. All conditions use the same token budget per prompt.

Baselines

Naive reference point. Zero-shot prompting with the task instruction and schema, and random whole-example few-shot prompting from the seed pool at a matched token budget. These show how much any demonstrations help. They also show whether few-shot raises sensitivity on parsing as it did on Brittlebench. If our method does not reduce the brittleness share $S$ below the random few-shot level, it has failed its central sensitivity claim.

Similarity retrieval. The strongest in-context competitor on accuracy is retrieval of the most similar whole examples from the seed pool by embedding similarity. This is the KATE-style baseline that Curriculum Demonstration Selection compared against. It uses the same pool, the same token budget, and an embedding model tuned on nothing. Similarity retrieval is a real threat on SQUALL, where questions about similar tables share structure. Beating it means higher accuracy on SMCALFLOW-CS and CFQ Recursive at matched budget, with a lower brittleness share on all three datasets.

Structure-ordered whole examples. The strongest existing curriculum method, reimplemented for parsing. We take PSL-ICL's idea of retrieving examples whose operation sequence is contained in the query's, ordered by step count. Here programs serve as the logic, and a first-pass zero-shot parse of the query predicts its operations. This replaces PSL-ICL's BREAK-trained analyzer, so the two methods share the same supervision. The reported reference is PSL-ICL's 2.24% average accuracy gain over prior in-context methods on Llama3-8B, using 2.74 examples on average. It was measured on reasoning question answering, and no reported result was found on Lacuna for PSL-ICL on semantic parsing. Beating this baseline isolates the value of span grounding and fragment granularity over structure-aware ordering alone.

Parameter-update ceiling. Span-level supervised attention fine-tuning on the same $k$ aligned examples, following Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention. It is the method the alignment literature would recommend. That paper reports a 13-point absolute gain at 16 shots on SMCALFLOW-CS. No absolute accuracy for that setting was found on Lacuna, so we rerun it ourselves with its BERT2SEQ setup and a matched hyperparameter search of 20 trials. For SQUALL, the published reference is 54.1% execution accuracy with BERT, against a 48.8% weakly supervised prior best. That figure uses alignment for the full training set, so we treat it as a ceiling, not a target. Matching the 16-shot fine-tuned parser without training would already be a meaningful result.

Ablations isolating the key ingredient. Four ablations each remove one piece:

  • Annotated whole examples: whole seed examples with their alignments written inline as markup, but no fragmenting. This tests whether the alignment information alone suffices.
  • Shuffled fragments: fragments in random order. This tests the role of order.
  • Non-nested fragments: fragments of the same depths drawn from unrelated examples. This tests the role of nesting.
  • Automatic alignments: fragments built from GIZA-style or LLM-induced alignments instead of gold. This tests whether gold alignment matters, as the low-resource alignment literature predicts it should at small $k$.

Why not just…?

Why not just use similarity-retrieved whole examples? Retrieval finds examples that look like the query, but looking alike does not guarantee the same structure. PSL-ICL reports that lexically similar questions often need different reasoning, and Curriculum Demonstration Selection beat KATE most on hard MATH problems. We give retrieval its best shot at an identical budget. If it matches us in both accuracy and brittleness share on CFQ Recursive, the fragment idea is not needed and we will say so.

Why not just add chain-of-thought or let the model decompose the query itself? Brittlebench found chain-of-thought cuts the perturbation-induced drop by only about 17%. Self-generated simplified sub-problems, as in Luo et al., are ungrounded and can go wrong. Still, a model writing its own sub-problems is the most plausible way to get our benefit without any annotation. We include a self-decomposition condition at matched output length. If it closes most of the gap, the conclusion is that structure matters and gold alignment does not. That is a useful negative result for the alignment literature.

Why not just fine-tune with the alignments, as the alignment literature does? Fine-tuning may well win on raw accuracy at $k = 64$, and we report it as a ceiling. The in-context route matters where weights cannot be touched or the seed pool changes per user. Our sensitivity question also has no analogue for a fine-tuned small parser. The comparison of interest is accuracy per annotated example, and how quickly each approach's accuracy grows with $k$.

Why not just average over prompts with self-consistency or sensitivity-based abstention? Majority voting over sampled answers, and abstaining when answers vary across prompts as in SENSEL, both treat brittleness after the fact at several times the inference cost. We report self-consistency at five samples as a cost-matched comparison. Our method should lower the brittleness share with a single forward pass. If it does not, voting is the better engineering answer.

Decisive Checks

The proposal rests on four outcomes, all measured on Llama3-8B and the 70B-class model at $k = 16$ and $k = 64$ across SQUALL, SMCALFLOW-CS, and CFQ Recursive.

  1. Accuracy. The fragment curriculum should exceed both similarity retrieval and structure-ordered whole examples, with the margin growing on CFQ Recursive's deeply nested queries.
  2. Sensitivity. The brittleness share $S$ should fall relative to random and similarity-retrieved few-shot prompts. This should hold under execution-based scoring, so the gain cannot be the formatting artifact identified by Flaw or Artifact. Gold-program log-probability should rise on the covered parts of the query.
  3. Order and nesting. Shuffling should hurt nested fragment prompts but not non-nested ones of equal depth composition. This interaction would explain why PSL-ICL found order critical and Curriculum Demonstration Selection did not.
  4. Gold alignment. Gold-aligned fragments should beat automatically aligned ones at small $k$, with the gap closing as $k$ grows.

The proposal is weakened in specific ways depending on which result fails. If annotated whole examples match fragments, fragmentation is unnecessary. If shuffled and ordered fragments tie in the nested condition, the curriculum explanation fails. If automatic alignments tie gold, the low-resource supervision story does not carry over to prompting.