Compositional Generalization for Primitive Substitutions

Yuanpeng Li, Liang Zhao, Jianyu Wang, Joel Hestness

TL;DR

This paper tackles the challenge of compositional generalization in neural networks, a key aspect of human language understanding that current models struggle with. They propose a novel approach using two distinct representations of input and reducing their entropy, leading to significant improvements in various NLP tasks, including outperforming human performance on a few-shot learning task.

What Problem Does This Paper Address?

Humans effortlessly understand and generate novel combinations of known words and concepts, a skill known as compositional generalization. For instance, knowing "step" and "jump" allows us to understand "jump twice." However, current neural networks, which typically rely on a single representation for an entire input sentence, fail to capture this systematicity. This leads to brittle models that perform poorly when encountering new combinations of familiar elements, especially when those elements weren't seen together during training. The paper aims to bridge this gap by equipping neural networks with a more human-like ability to generalize compositionally.

Methods

The core of the proposed method lies in leveraging prior knowledge of compositionality through two distinct representations of the input sentence:

  1. Primitive Representation: This representation focuses on the core meaning or "what" of the input words.
  2. Function Representation: This representation captures the "how" or the structural/functional aspect of the input.

Instead of a single representation, the model learns these two, and crucially, it reduces the entropy of each representation. This is achieved by adding noise during training and regularizing the L2 norm of the representations, effectively making them more focused and less noisy.

The model then uses an attention mechanism to combine these representations. The function representation is used to generate attention maps over the primitive representation. Each attention map helps select relevant parts of the primitive representation to construct the output sequence of actions. This separation ensures that output action types are primarily influenced by the primitive representation, while output action order is guided by the function representation.

Datasets

The paper evaluates its approach on several datasets:

  • SCAN dataset: A benchmark for compositional generalization, specifically focusing on the "Primitive tasks" (Jump and TurnLeft) and "Template-matching tasks."
  • SCAN-ADJ: An extension of the SCAN dataset designed to test compositional generalization when words carry both primitive and functional information.
  • Few-shot Learning Task dataset: A dataset from Lake et al. (2019) used to evaluate the model's ability to learn from very limited examples.
  • Machine Translation dataset: An English-French translation task to assess applicability to other sequence-to-sequence problems.

Key Insights and Ideas

  • Decomposition is Key: The paper strongly suggests that breaking down input into distinct "what" (primitive) and "how" (functional) representations is crucial for achieving compositional generalization, moving beyond single, holistic sentence representations.
  • Entropy Reduction for Focus: Reducing the entropy of these representations, by adding controlled noise and regularization, forces the model to focus on the essential information within each representation, leading to better generalization.
  • Attention as a Bridge: The attention mechanism acts as a powerful tool to selectively combine information from the primitive and functional representations, allowing the model to map input components to output actions in a structured way.
  • Human-Level Few-Shot Learning: The most striking insight is the model's ability to outperform human performance on a few-shot learning task, challenging the notion that humans inherently possess superior learning capabilities from limited data.

Key Results

The proposed method demonstrates significant improvements across various tasks:

  • SCAN Primitive Tasks: Achieved accuracies of 98.8% on the Jump task (up from 14.0%) and 99.7% on the TurnLeft task (up from 92.0%).
  • SCAN Template-matching: Showed significantly higher performance than baselines across four tasks.
  • SCAN-ADJ: Achieved 100.0% accuracy, compared to a standard LSTM's 2.1%.
  • Few-shot Learning Task: Outperformed human performance, achieving 95.0% ± 6.8% on the Primitive task, compared to humans at 82.8%.
  • Machine Translation: Increased sentence accuracy from 12.5% to 62.5%, with an adjusted accuracy of 100.0% when considering semantically correct but not verbatim translations.

Limitations and Future Work

  • SCAN Dataset Limitations: The paper notes that the SCAN dataset has a limitation where words typically contain either primitive or functional information, but not both. They introduced SCAN-ADJ to address this, but future work could explore more complex real-world scenarios where words inherently blend these aspects.
  • Generalizability to Other Tasks: While promising results were shown on machine translation, further exploration is needed to fully assess the approach's applicability to a wider range of sequence-to-sequence tasks.
  • Computational Cost: The paper doesn't explicitly detail computational costs, but the use of two representations and attention mechanisms might imply higher computational demands compared to simpler models.
  • Future Directions: The authors hope their work will pave the way for human-level compositional language learning, encouraging machines to learn quickly and generalize widely, making them more helpful in various tasks. They also suggest that compositionality is relevant to multimodal problems like image captioning and visual question answering.

Summary updated

Abstract

Compositional generalization is a basic mechanism in human language learning, but current neural networks lack such ability. In this paper, we conduct fundamental research for encoding compositionality in neural networks. Conventional methods use a single representation for the input sentence, making it hard to apply prior knowledge of compositionality. In contrast, our approach leverages such knowledge with two representations, one generating attention maps, and the other mapping attended input words to output symbols. We reduce the entropy in each representation to improve generalization. Our experiments demonstrate significant improvements over the conventional methods in five NLP tasks including instruction learning and machine translation. In the SCAN domain, it boosts accuracies from 14.0% to 98.8% in Jump task, and from 92.0% to 99.7% in TurnLeft task. It also beats human performance on a few-shot learning task. We hope the proposed approach can help ease future research towards human-level compositional language learning.
BibTeX
@inproceedings{li2019compositional,
  author = {Yuanpeng Li and Liang Zhao and Jianyu Wang and Joel Hestness},
  title = {{Compositional Generalization for Primitive Substitutions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=yV9cKKTNwi},
}
Figure