Meta-Learning to Compositionally Generalize

Henry Conklin, Bailin Wang, Kenny Smith, Ivan Titov

Abstract

Neural networks often struggle with compositional generalization, preferring to memorize training sequences rather than learning underlying rules. This paper presents a meta-learning framework that uses "virtual tasks" composed of similar examples to penalize memorization. By forcing models to perform well on related but non-identical tasks, the approach encourages the learning of atomic components and structural rules. Evaluated on SCAN and COGS benchmarks, the method significantly outperforms standard supervised baselines across different architectures.

Introduction

Human language is inherently compositional: we understand "The purple elephant danced" even if we’ve never seen those words together, because we know what the individual parts mean and how they combine. Neural networks, however, often take the "easy way out." Instead of learning these underlying rules, they frequently memorize large chunks of training data.

While this strategy works for standard (i.i.d.) testing where test data mirrors training data, it fails in out-of-distribution (OOD) scenarios. In these cases, models must handle familiar words in entirely new combinations. The authors argue that standard supervised learning lacks the incentives for compositionality, as it doesn't distinguish between a model that understands and one that simply remembers.

Methodology: The Virtual Task Strategy

The authors adapt the DG-MAML (Domain Generalization with Model-Agnostic Meta-Learning) algorithm to address this. The core idea is to shift the training objective from minimizing error on a single batch to updating the model so it performs well on a related, novel task.

For every training step, the framework creates a pair of tasks:

  1. Meta-Train: A random batch of training data.
  2. Meta-Test: A batch of examples that are similar to the meta-train batch, but not identical.

To identify these "similar" examples, three metrics are employed:

  • Levenshtein Distance: Measures the number of edits required to transform one sentence into another.
  • String Kernels: Identifies common subsequences of words.
  • Tree Kernels: Used primarily for the COGS dataset, these compare the structural similarity of logical forms and dependency trees.
Figure 2: Examples of sentences, their logical forms, and the dependency/partial tree structures used to calculate structural similarity in the COGS dataset.

By training on tasks populated with similar examples (e.g., "The cat gave a gift" and "The dog gave a gift"), the model is discouraged from memorizing sentences as single blocks. If it attempts to memorize, the gradient update from the first sentence will not assist in solving the second. Success requires the model to learn the "atoms"—individual words and the rules for their combination.

Experimental Benchmarks

The method was evaluated on two primary benchmarks designed to test compositional generalization:

  • SCAN: A dataset of navigation commands (e.g., "walk twice and turn left") mapped to action sequences. The researchers utilized Maximum Compound Divergence (MCD) splits, which maximize the difference between training and test sets.
  • COGS: A complex semantic parsing dataset mapping natural language to logical forms. It tests "productivity" (handling longer sentences) and "systematicity" (using known verbs with new subjects).

Key Results and Insights

The meta-learning approach demonstrated significant improvements over standard supervised baselines:

  • SCAN (MCD1 split): The Lev-MAML + LSTM model achieved 58.9% accuracy, more than doubling the performance of the standard LSTM baseline (27.4%). It notably outperformed larger pre-trained models like T5-base.
  • COGS: The Tree-MAML + Transformer reached 66.7% accuracy on the generalization set, compared to 58.6% for a standard Transformer.
  • Architecture Agnostic: Improvements were consistent across both LSTMs and Transformers, indicating the method's broad applicability.

Key insights include the realization that memorization is often a "choice" driven by training objectives rather than a lack of capacity. Furthermore, structural similarity (Tree-MAML) proved most effective for complex datasets like COGS, suggesting that providing hints about underlying syntax is highly beneficial.

Limitations and Future Work

While rigorous, the SCAN and COGS datasets are synthetic and do not capture the full ambiguity of natural language. Future work will focus on testing the framework on more "natural" data. Additionally, the computational cost of calculating similarity ($O(N^2)$) remains a challenge for scaling to massive datasets. Finally, developing more universal, unsupervised similarity metrics is a priority, as tree kernels require structured data that is not always available.

Summary updated

Abstract

Natural language is compositional; the meaning of a sentence is a function of the meaning of its parts. This property allows humans to create and interpret novel sentences, generalizing robustly outside their prior experience. Neural networks have been shown to struggle with this kind of generalization, in particular performing poorly on tasks designed to assess compositional generalization (i.e. where training and testing distributions differ in ways that would be trivial for a compositional strategy to resolve). Their poor performance on these tasks may in part be due to the nature of supervised learning which assumes training and testing data to be drawn from the same distribution. We implement a meta-learning augmented version of supervised learning whose objective directly optimizes for out-of-distribution generalization. We construct pairs of tasks for meta-learning by sub-sampling existing training data. Each pair of tasks is constructed to contain relevant examples, as determined by a similarity metric, in an effort to inhibit models from memorizing their input. Experimental results on the COGS and SCAN datasets show that our similarity-driven meta-learning can improve generalization performance.
BibTeX
@inproceedings{conklin2021meta,
  author = {Henry Conklin and Bailin Wang and Kenny Smith and Ivan Titov},
  title = {{Meta-Learning to Compositionally Generalize}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.258},
}
Figure