Improving Compositional Generalization with Latent Structure and Data Augmentation

Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Krzysztof Nowak, Tal Linzen, Fei Sha, Kristina Toutanova

Abstract

Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization. Compositional data augmentation via example recombination has transferred some prior knowledge about compositionality to such black-box neural models for several semantic parsing tasks, but this often required task-specific engineering or provided limited gains. We present a more powerful data recombination method using a model called Compositional Structure Learner (CSL). CSL is a generative model with a quasi-synchronous context-free grammar backbone, which we induce from the training data. We sample recombined examples from CSL and add them to the fine-tuning data of a pre-trained sequence-to-sequence model (T5). This procedure effectively transfers most of CSL’s compositional bias to T5 for diagnostic tasks, and results in a model even stronger than a T5-CSL ensemble on two real-world compositional generalization tasks. This results in new state-of-the-art performance for these challenging semantic parsing tasks requiring generalization to both natural language variation and novel compositions of elements.

Introduction

Modern neural networks, particularly Transformers like T5, have revolutionized natural language processing. However, they possess a significant Achilles' heel: compositional generalization. This is the ability to understand and produce novel combinations of familiar components—a skill humans use effortlessly. For instance, if a person understands the command "jump" and the modifier "twice," they can immediately interpret "jump twice" even if they have never encountered that specific phrase. Standard neural models often fail these "out-of-distribution" tests, struggling to map natural language to structured programs when the specific combination is novel.

Methodology: The Compositional Structure Learner (CSL)

To address this, we propose a two-stage pipeline that combines the strengths of symbolic grammar-based models with the flexibility of large-scale neural networks. The core of this approach is the Compositional Structure Learner (CSL).

Overview of the CSL data augmentation pipeline, showing the transition from original examples to a grammar-induced generative model and finally to an augmented T5 model.

As illustrated in the figure above, the process involves:

  1. Grammar Induction: Instead of manually writing rules, CSL automatically induces a Quasi-Synchronous Context-Free Grammar (QCFG) from the training data. It identifies the most efficient rules to explain how input sentences map to output programs using a Minimum Description Length objective.
  2. Data Augmentation: As a generative model, CSL can sample new, never-before-seen pairs of utterances and programs that follow the induced rules. We generate 100,000 such synthetic examples.
  3. Fine-tuning T5: A standard T5 model is fine-tuned on a combination of the original training data and these synthetic examples. This process effectively "transfers" the compositional inductive bias of the grammar into the high-capacity T5 model.

Experimental Evaluation

We evaluated the CSL-augmented T5 model across four major benchmarks designed to test different facets of compositional generalization:

  • SCAN & COGS: Synthetic benchmarks testing specific generalization types, such as handling longer sequences or novel verb-noun combinations.
  • GeoQuery: A real-world dataset of US geography questions mapped to database queries.
  • SMCalFlow-CS: A complex, large-scale dataset for calendar and organizational tasks involving significant natural language variation.

Key Results

The results demonstrate that CSL-Augmentation provides a substantial boost over standard T5 and previous state-of-the-art (SOTA) methods.

Performance comparison showing that T5+CSL-Aug. achieves near-perfect scores on synthetic tasks and sets new SOTA records on GeoQuery and SMCalFlow-CS.

On the diagnostic SCAN and COGS benchmarks, the augmented model reached near-perfect accuracy (up to 99.7%), effectively solving tasks where standard T5 models often struggle. More importantly, on real-world datasets like GeoQuery (Length split), accuracy jumped from 40.0% to 67.8%. On SMCalFlow-CS, the model consistently outperformed previous specialized architectures like C2F+SS, particularly in the most challenging "32-C" split, reaching 70.4% accuracy.

Discussion and Insights

A critical finding of this work is the "best of both worlds" synergy. While symbolic models like CSL are excellent at composition, they often lack the natural language coverage of LLMs. Conversely, LLMs excel at language variety but fail at composition. By using CSL as a "teacher" via data augmentation, the T5 model learns robust structural principles.

Interestingly, the augmented T5 model often performed well even on test cases that the induced grammar itself could not generate. This suggests that the model is not merely memorizing synthetic examples but is developing a more generalized understanding of linguistic structure. Furthermore, CSL is computationally efficient, reducing grammar induction runtime by over 90% compared to previous methods like NQG, making it viable for large-scale applications.

Conclusion

By combining latent structure induction with large-scale data augmentation, we have shown that it is possible to significantly improve the compositional generalization of standard sequence-to-sequence models. This approach sets new performance standards on several challenging benchmarks and provides a scalable path toward more robust and human-like natural language understanding.

Summary updated

Abstract

Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization. Compositional data augmentation via example recombination has transferred some prior knowledge about compositionality to such black-box neural models for several semantic parsing tasks, but this often required task-specific engineering or provided limited gains. We present a more powerful data recombination method using a model called Compositional Structure Learner (CSL). CSL is a generative model with a quasi-synchronous context-free grammar backbone, which we induce from the training data. We sample recombined examples from CSL and add them to the fine-tuning data of a pre-trained sequence-to-sequence model (T5). This procedure effectively transfers most of CSL’s compositional bias to T5 for diagnostic tasks, and results in a model even stronger than a T5-CSL ensemble on two real world compositional generalization tasks. This results in new state-of-the-art performance for these challenging semantic parsing tasks requiring generalization to both natural language variation and novel compositions of elements.
BibTeX
@inproceedings{qiu2022improving,
  author = {Linlu Qiu and Peter Shaw and Panupong Pasupat and Pawel Krzysztof Nowak and Tal Linzen and Fei Sha and Kristina Toutanova},
  title = {{Improving Compositional Generalization with Latent Structure and Data Augmentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=3o3Cde3Rxm},
}
Figure