Abstract
Inbar Oren, Jonathan Herzig, Jonathan Berant. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
2021.
Introduction
Semantic parsing—the task of mapping natural language (e.g., "Find me a hotel with a gym") into machine-executable code (e.g., SQL)—is a cornerstone of modern AI interfaces. However, these models often struggle with compositional generalization: the ability to combine known components in novel ways. While a model might understand "hotels in Tokyo" and "restaurants with WiFi," it may fail on "hotels in Tokyo with WiFi." This "compositional gap" highlights a fundamental limitation in how models learn from data, often favoring memorization over true reasoning.
Methodology: Smart Sampling for Structural Diversity
To address this, researchers propose that the quality of training data, specifically its structural diversity, is more important than its sheer volume. When using large-scale synthetic data generated by grammars, the resulting "haystack" is often redundant. The authors introduce two primary sampling strategies to find the "needles"—examples that maximize structural variety:
- Uniform Abstract Template (UAT) Sampling: Queries are grouped into templates by abstracting specific entities (e.g., replacing "Tokyo" with a generic
CITYtag). Sampling is then performed uniformly across these templates, ensuring rare structures are well-represented. - Compound Maximum Entropy (CMAXENT) Sampling: This approach focuses on "compounds," or small sub-trees within the logical form. It uses a greedy algorithm to select examples that maximize the entropy (diversity) of these sub-structures in the training set.
Datasets and the Compositional Split
The study utilizes the Schema2QA dataset across six domains. A key innovation is the introduction of a Compositional Split. Unlike standard independent and identically distributed (i.i.d.) splits, where training and test data share similar structures, the compositional split ensures that the logical structures (templates) encountered at test time were never seen during training. This setup forces the model to generalize its understanding of language components rather than relying on memorized patterns.
Key Results and Efficiency
The results demonstrate a massive efficiency gain. On the challenging compositional split, a baseline model scored 20.1%. Training on 1 million randomly sampled synthetic examples improved this to 37.7%. Remarkably, using UAT sampling with only 5,000 examples—just 0.5% of the data—achieved a superior score of 41.4%. This 200x increase in data efficiency proves that structural variety is the "secret sauce" for generalization.
Furthermore, diverse sampling helps break "co-occurrence biases." In biased datasets, certain properties often appear together, leading models to treat them as a single unit. By forcing the model to see components in varied contexts, it learns their independent meanings.
Conclusion
This research underscores that for complex reasoning tasks, more data is not always better. By strategically selecting a small, structurally diverse subset of synthetic data, models can achieve significantly better compositional generalization with a fraction of the computational cost. Future work will look into reducing dependency on high-quality grammars and improving entity linking.
Summary updated