Abstract
Modern semantic parsers suffer from two principal limitations. First, training requires expensive collection of utterance-program pairs. Second, semantic parsers fail to generalize at test time to new compositions/structures that have not been observed during training. Recent research has shown that automatic generation of synthetic utterance-program pairs can alleviate the first problem, but its potential for the second has thus far been under-explored. In this work, we investigate automatic generation of synthetic utterance-program pairs for improving compositional generalization in semantic parsing. Given a small training set of annotated examples and an "infinite" pool of synthetic examples, we select a subset of synthetic examples that are structurally-diverse and use them to improve compositional generalization. We evaluate our approach on a new split of the schema2QA dataset, and show that it leads to dramatic improvements in compositional generalization as well as moderate improvements in the traditional i.i.d setup. Moreover, structurally-diverse sampling achieves these improvements with as few as 5K examples, compared to 1M examples when sampling uniformly at random -- a 200x improvement in data efficiency.
Introduction
Semantic parsers are the engines behind virtual assistants, mapping natural language queries (e.g., "Find me a pet-friendly hotel") into executable code. While current models perform well on queries similar to their training data, they often fail at compositional generalization—the ability to understand and execute novel combinations of known components. For instance, a model might understand "show me Thai restaurants" and "how many hotels are in Tokyo," yet fail to process "how many Thai restaurants are in Tokyo?" This paper demonstrates that the key to overcoming this bottleneck lies not in the quantity of synthetic training data, but in its structural diversity.
Methodology
The researchers propose a shift from uniform random sampling to structurally-diverse sampling. By strategically selecting synthetic examples, they aim to expose the model to a wide array of logical structures, preventing it from over-fitting to simple, common patterns.
Two primary sampling strategies are introduced:
- Uniform Abstract Template (UAT) Sampling: Queries are converted into abstract templates by replacing specific database constants with generic tokens (e.g.,
entity,property). By sampling these templates uniformly, the model is forced to learn rare and complex structures as frequently as simple ones. - Compound Maximum Entropy (CMaxEnt) Sampling: This method focuses on "compounds" (sub-trees of the query structure). It uses a greedy heuristic to select examples that maximize the overall diversity (entropy) of these compounds in the training set.
Experimental Setup and Datasets
The evaluation uses the Schema2QA dataset across six domains. To measure compositional generalization effectively, the authors created a new compositional split. Unlike standard i.i.d. (independent and identically distributed) splits, this split ensures that the logical structures (templates) seen during testing were never encountered during training.
Key Results and Insights
The study reveals that structural diversity is a powerful driver of data efficiency.
- 200x Efficiency Gain: UAT sampling achieved peak performance with just 5,000 examples. In contrast, uniform random sampling required 1 million examples to approach similar performance levels.
- Accuracy Improvements: On the compositional test set, UAT sampling with 5K examples reached 41.4% accuracy, significantly outperforming the baseline of 20.1% and even the massive 1M-example uniform sampling at 37.7%.
- I.I.D. Stability: Optimizing for compositional generalization did not degrade performance on standard, simpler queries; i.i.d. performance remained stable or slightly improved.
Conclusion and Future Work
The findings suggest that the distribution of structures in training data is more important than sheer volume for building robust semantic parsers. While the model showed significant gains in structural understanding, it experienced a slight increase in "linking errors" (e.g., hallucinating entities). Future research will focus on refining the definition of structural units and improving the naturalness of synthetic language to further bridge the gap between grammar-generated data and human utterances.
Summary updated