Abstract
Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization. Compositional data augmentation via example recombination has transferred some prior knowledge about compositionality to such black-box neural models for several semantic parsing tasks, but this often required task-specific engineering or provided limited gains. We present a more powerful data recombination method using a model called Compositional Structure Learner (CSL). CSL is a generative model with a quasi-synchronous context-free grammar backbone, which we induce from the training data. We sample recombined examples from CSL and add them to the fine-tuning data of a pre-trained sequence-to-sequence model (T5). This procedure effectively transfers most of CSL's compositional bias to T5 for diagnostic tasks, and results in a model even stronger than a T5-CSL ensemble on two real world compositional generalization tasks. This results in new state-of-the-art performance for these challenging semantic parsing tasks requiring generalization to both natural language variation and novel compositions of elements.
Introduction: The Challenge of Compositional Generalization
Standard neural networks excel at pattern matching but often struggle with compositional generalization: the ability to understand novel combinations of known "atoms." For instance, a model might understand "jump" and "twice" individually but fail to interpret "jump twice" if that specific sequence was absent from training. While specialized architectures exist, they often lack the flexibility to handle the nuances of natural language. Conversely, large pre-trained models like T5 handle linguistic variation well but lack the structural discipline for out-of-distribution combinations. This research bridges the gap by using a structured generative model to "teach" a flexible sequence-to-sequence model the underlying rules of a task through data augmentation.
Methods: The Compositional Structure Learner (CSL)
The authors introduce the Compositional Structure Learner (CSL), a generative model with a Quasi-Synchronous Context-Free Grammar (QCFG) backbone induced automatically from training data.
The process involves two main stages:
- Grammar Induction: The model searches for rules that balance data fit with simplicity using a Minimum Description Length objective, allowing it to discover recursive structures (e.g., "X and Y") without manual engineering.
- Probabilistic Modeling: CSL learns the probabilities of applying specific rules in context using latent states.
After training, synthetic examples representing novel but valid combinations are sampled from CSL. A standard T5 model is then fine-tuned on a mixture of original and synthetic data, distilling the compositional bias of the grammar-based CSL into the high-capacity T5 model.
Evaluation and Datasets
The method was evaluated on both synthetic benchmarks and real-world natural language tasks:
- SCAN & COGS: Designed to test navigation and semantic parsing under distribution shifts (e.g., longer sequences or new primitive combinations).
- GeoQuery: A geography question dataset using "compositional splits" (Template, TMCD, and Length) to test structural generalization.
- SMCalFlow-CS: A complex task involving calendar and organizational queries, featuring natural language variation and cross-domain challenges.
Key Results
The T5+CSL-Augmentation method achieved state-of-the-art (SOTA) performance across multiple benchmarks.
- SOTA Performance: The method reached near 100% accuracy on SCAN and COGS.
- Real-World Gains: On the GeoQuery Length split, performance improved from 40.0% (base T5) to 67.8%.
- SMCalFlow-CS: In the challenging "8-shot" cross-domain split, accuracy rose from 34.7% to 51.6%, establishing a new SOTA.
- Comparison to GECA: CSL consistently outperformed GECA, as CSL's recursive recombination capabilities exceed GECA's fixed-window approach.
Discussion and Insights
- Bias Transfer: The results demonstrate that specialized architectures aren't always necessary; structured "teacher" models can effectively transfer structural biases to generic models via data.
- Generalization Beyond Grammar: Surprisingly, T5 improved even on test inputs that the CSL grammar could not parse, suggesting the synthetic data helps T5 learn a broader "compositional spirit."
- Scalability: CSL is more computationally efficient than predecessors like NQG, enabling grammar induction on larger datasets like SMCalFlow-CS.
Limitations
Despite its success, the QCFG formalism struggles with certain linguistic phenomena like logical variables or complex distributive properties. Additionally, on complex datasets like SMCalFlow, the induced grammar covered only about 13% of utterances. Improving grammar coverage for natural language variation remains a key area for future work.
Summary updated