Align and Augment: Generative Data Augmentation for Compositional Generalization

Francesco Cazzaro, Davide Locatelli, Ariadna QuattoniEACL 2024

TL;DR

This paper introduces ARCHER, a novel data augmentation technique that leverages word alignment information to improve compositional generalization in semantic parsing. By training a two-step generative model that first creates monotonically aligned sequences and then reorders them, ARCHER significantly boosts the performance of seq2seq models on challenging generalization tasks, particularly those involving longer sequences.

What Problem Does This Paper Address?

Semantic parsing aims to convert natural language sentences into structured meaning representations (MRs). While sequence-to-sequence (seq2seq) models have achieved impressive performance, they notoriously struggle with compositional generalization. This means they fail to correctly interpret novel combinations of familiar words and structures that were not explicitly seen during training. For instance, a model might understand "flights to Boston" and "flights from New York," but struggle with "flights from Boston to New York" if it hasn't encountered such a structure. This limitation hinders their real-world applicability. Data augmentation, which artificially expands the training set, is a promising avenue to address this, but existing methods often fall short in improving compositional generalization.

Methods

ARCHER tackles compositional generalization through a two-step generative process that relies on word alignment annotations between natural language (NL) sentences and their corresponding MRs.

  1. Generator: This step first creates monotonically aligned NL/MR pairs. This involves reordering the MR symbols to align directly with the NL words. A generative model, specifically a Weighted Finite Automaton (WFA), is trained on these monotonically aligned bi-symbol sequences (pairs of NL and MR tokens). The WFA learns the distribution of these aligned sequences and can then generate new, unseen bi-symbol sequences.
  2. Reorderer: The generated bi-symbol sequences from the Generator are then passed to a Reorderer model. This model takes the unordered MR symbols and reorders them to match the original NL sentence's structure. The Reorderer is trained on pairs of original MRs and their monotonically aligned versions. Crucially, the Reorderer is unconstrained, allowing it to correct errors beyond simple reordering, which proved beneficial.

The augmented data, consisting of these generated and reordered NL/MR pairs, is then added to the original training set to train the semantic parser.

Datasets

The authors evaluate ARCHER on two multilingual semantic parsing benchmarks:

  • GEOALIGNED: An extension of the GEO dataset, annotated with word alignments.
  • ATISALIGNED: A newly introduced benchmark, extending the ATIS dataset with word alignment annotations.

Both datasets are used for evaluating compositional generalization, particularly through "query" and "length" partitions.

Key Insights and Ideas

  • Word alignments are crucial for compositional generalization: The paper strongly suggests that explicitly leveraging word alignments between NL and MR is key to improving how models handle novel combinations.
  • Two-step generation is effective: Separating the generation of aligned sequences from the reordering step allows for both modeling complex sequence distributions (Generator) and correcting structural errors (Reorderer).
  • Unconstrained reordering offers benefits: Allowing the Reorderer to freely modify MRs (add, delete, substitute) beyond simple permutation helped fix errors made by the Generator, leading to better overall quality.
  • WFAs are well-suited for density estimation: The choice of WFAs for modeling the bi-symbol distribution is justified by their ability to perform density estimation and model prefix distributions, which is essential for generative tasks.
  • Filtering is necessary but requires balance: While filtering generated samples improves quality, selecting only the most probable samples can reduce diversity, highlighting a trade-off between correctness and diversity.

Key Results

ARCHER significantly improves compositional generalization performance, especially on the challenging "length partition" (where models are tested on longer sequences than seen during training).

  • On the English GEO dataset's length partition, ARCHER nearly doubles parser accuracy to 46%.
  • In the query partition, performance improves from 72% to 82%.
  • ARCHER consistently outperforms other data augmentation techniques like GECA and Self-Training, particularly on length-based generalization tasks.
  • An analysis of generated data shows ARCHER produces samples that are both more correct and more diverse than competing methods.

Limitations and Future Work

  • Computational Cost: ARCHER is computationally demanding, requiring training multiple models. This is acknowledged as a limitation compared to simpler methods like GECA.
  • MR Formalism: The current work focuses on the FunQL formalism for MRs. Future work could explore applying ARCHER to other formalisms, though this is limited by the availability of word alignment annotations.
  • Filtering Trade-off: The paper notes the inherent trade-off between sample diversity and correctness when filtering augmented data. Future work could explore more sophisticated filtering strategies.
  • Exploring other density estimators: While WFAs were effective, the authors suggest exploring other density estimation methods for modeling bi-symbol sequences.

Summary updated

BibTeX
@inproceedings{cazzaro2024align,
  author = {Francesco Cazzaro and Davide Locatelli and Ariadna Quattoni},
  title = {{Align and Augment: Generative Data Augmentation for Compositional Generalization}},
  booktitle = {EACL},
  year = {2024},
}
Figure