Abstract
Natural language is compositional; the meaning of a sentence is a function of the meaning of its parts. This property allows humans to create and interpret novel sentences, generalizing robustly outside their prior experience. Neural networks have been shown to struggle with this kind of generalization, in particular performing poorly on tasks designed to assess compositional generalization (i.e. where training and testing distributions differ in ways that would be trivial for a compositional strategy to resolve). Their poor performance on these tasks may in part be due to the nature of supervised learning which assumes training and testing data to be drawn from the same distribution. We implement a meta-learning augmented version of supervised learning whose objective directly optimizes for out-of-distribution generalization. We construct pairs of tasks for meta-learning by sub-sampling existing training data. Each pair of tasks is constructed to contain relevant examples, as determined by a similarity metric, in an effort to inhibit models from memorizing their input. Experimental results on the COGS and SCAN datasets show that our similarity-driven meta-learning can improve generalization performance.
Introduction
Human language is inherently compositional: if you know what "red" means and what "square" means, you instantly understand "red square" even if you’ve never seen one. Standard neural networks, however, are notorious for "cheating." Instead of learning the underlying rules of composition, they often memorize large chunks of training data.
This leads to a massive performance drop when the model encounters out-of-distribution (OOD) data—scenarios where the individual components are familiar, but their combination is new (e.g., seeing "the cat gave the dog a gift" in training and being tested on "the dog gave the cat a gift"). The authors argue that the standard supervised learning objective (minimizing loss on a single distribution) doesn't provide the right incentives for models to learn these robust, atomic rules.
Methods: Similarity-Driven Meta-Learning
The authors implement a variant of Model-Agnostic Meta-Learning (MAML), specifically Domain Generalization MAML (DG-MAML). The core idea is to simulate the "distribution shift" during training. Instead of training on one big batch, they create "virtual tasks" consisting of a meta-train set and a meta-test set.
The key innovation is the use of Similarity Metrics to find examples that share "atoms" (words or structures) but differ in arrangement. They use three metrics:
- Levenshtein Distance: Based on word-level edit distance.
- String-Kernel Similarity: Based on shared subsequences.
- Tree-Kernel Similarity: Based on shared syntactic sub-structures using the logical form of the sentence.
The model takes a gradient step on the meta-train set. The final loss, however, is calculated based on how well that updated model performs on the meta-test set. This forces the model to prioritize updates that are "compositionally useful" across different but related examples, effectively punishing the memorization of specific phrases.
Experimental Results
The researchers evaluated their method on two major compositional generalization benchmarks:
- SCAN: A dataset of navigation commands (e.g., "walk twice and turn left"). On the toughest Maximum Compound Divergence (MCD) splits, Lev-MAML significantly improved the LSTM baseline (e.g., on MCD1, accuracy rose from 27.4% to 47.6%).
- COGS: A diverse semantic parsing dataset. Tree-MAML provided a massive boost. For the LSTM baseline, accuracy on the generalization set jumped from 30.3% to 41.2%. For the Transformer baseline, it rose from 54.7% to 64.1%.
Discussion and Insights
Neural networks have the capacity to generalize, but they often "choose" to memorize because it's the path of least resistance in standard supervised learning. Meta-learning acts as a regularizer that makes memorization a losing strategy. Simply doing random meta-learning helps, but similarity-driven sampling is much more effective. By picking meta-test examples that are "near neighbors," the model is forced to reconcile the differences between two very similar inputs, which highlights the underlying atomic structure.
On the COGS dataset, Tree-MAML outperformed string-based methods. This suggests that providing the model with a "structural" bias during the meta-learning phase is highly beneficial for semantic tasks. Furthermore, the method is complementary to data augmentation techniques like GECA, suggesting that changing the objective and changing the data are two different, but compatible, ways to solve the problem.
Conclusion
This work provides a compelling argument that solving compositionality may not require entirely new architectures, but rather "tougher" training objectives. By meta-learning to generalize across similar but distinct structural patterns, models can be nudged away from memorization and toward the robust systematicity characteristic of human language.
Summary updated