Abstract
Neural networks often struggle with compositional generalization, preferring to memorize training sequences rather than learning underlying rules. This paper presents a meta-learning framework that uses "virtual tasks" composed of similar examples to penalize memorization. By forcing models to perform well on related but non-identical tasks, the approach encourages the learning of atomic components and structural rules. Evaluated on SCAN and COGS benchmarks, the method significantly outperforms standard supervised baselines across different architectures.
Introduction
Human language is inherently compositional: we understand "The purple elephant danced" even if we’ve never seen those words together, because we know what the individual parts mean and how they combine. Neural networks, however, often take the "easy way out." Instead of learning these underlying rules, they frequently memorize large chunks of training data.
While this strategy works for standard (i.i.d.) testing where test data mirrors training data, it fails in out-of-distribution (OOD) scenarios. In these cases, models must handle familiar words in entirely new combinations. The authors argue that standard supervised learning lacks the incentives for compositionality, as it doesn't distinguish between a model that understands and one that simply remembers.
Methodology: The Virtual Task Strategy
The authors adapt the DG-MAML (Domain Generalization with Model-Agnostic Meta-Learning) algorithm to address this. The core idea is to shift the training objective from minimizing error on a single batch to updating the model so it performs well on a related, novel task.
For every training step, the framework creates a pair of tasks:
- Meta-Train: A random batch of training data.
- Meta-Test: A batch of examples that are similar to the meta-train batch, but not identical.
To identify these "similar" examples, three metrics are employed:
- Levenshtein Distance: Measures the number of edits required to transform one sentence into another.
- String Kernels: Identifies common subsequences of words.
- Tree Kernels: Used primarily for the COGS dataset, these compare the structural similarity of logical forms and dependency trees.
By training on tasks populated with similar examples (e.g., "The cat gave a gift" and "The dog gave a gift"), the model is discouraged from memorizing sentences as single blocks. If it attempts to memorize, the gradient update from the first sentence will not assist in solving the second. Success requires the model to learn the "atoms"—individual words and the rules for their combination.
Experimental Benchmarks
The method was evaluated on two primary benchmarks designed to test compositional generalization:
- SCAN: A dataset of navigation commands (e.g., "walk twice and turn left") mapped to action sequences. The researchers utilized Maximum Compound Divergence (MCD) splits, which maximize the difference between training and test sets.
- COGS: A complex semantic parsing dataset mapping natural language to logical forms. It tests "productivity" (handling longer sentences) and "systematicity" (using known verbs with new subjects).
Key Results and Insights
The meta-learning approach demonstrated significant improvements over standard supervised baselines:
- SCAN (MCD1 split): The Lev-MAML + LSTM model achieved 58.9% accuracy, more than doubling the performance of the standard LSTM baseline (27.4%). It notably outperformed larger pre-trained models like T5-base.
- COGS: The Tree-MAML + Transformer reached 66.7% accuracy on the generalization set, compared to 58.6% for a standard Transformer.
- Architecture Agnostic: Improvements were consistent across both LSTMs and Transformers, indicating the method's broad applicability.
Key insights include the realization that memorization is often a "choice" driven by training objectives rather than a lack of capacity. Furthermore, structural similarity (Tree-MAML) proved most effective for complex datasets like COGS, suggesting that providing hints about underlying syntax is highly beneficial.
Limitations and Future Work
While rigorous, the SCAN and COGS datasets are synthetic and do not capture the full ambiguity of natural language. Future work will focus on testing the framework on more "natural" data. Additionally, the computational cost of calculating similarity ($O(N^2)$) remains a challenge for scaling to massive datasets. Finally, developing more universal, unsupervised similarity metrics is a priority, as tree kernels require structured data that is not always available.
Summary updated