Improving Compositional Generalization in Classification Tasks via Structure Annotations

Juyong Kim, Pradeep Ravikumar, Joshua Ainslie, Santiago Ontañón

TL;DR

Neural models struggle to generalize compositionally, meaning they can't reliably combine known concepts to understand new ones. This paper tackles this by showing that providing structural annotations, like parse trees, to Transformer models significantly improves their compositional generalization on a new classification dataset derived from semantic parsing.

What Problem Does This Paper Address?

Compositional generalization is a fundamental aspect of human intelligence: the ability to understand and generate novel combinations of familiar components. For instance, if you know "jump" and "twice," you can infer "jump twice." While humans excel at this, state-of-the-art neural networks, particularly in Natural Language Processing (NLP), often fail to generalize compositionally. They might perform well on data similar to their training set but falter when faced with new combinations of learned elements. This limitation hinders their real-world applicability in complex, dynamic environments. This paper focuses on improving this capability in classification tasks, which are simpler to evaluate than sequence generation tasks.

Methods

The core contribution of this work lies in two main areas:

  1. Dataset Creation: The authors convert the Compositional Freebase Questions (CFQ) dataset, originally designed for semantic parsing (translating natural language to SPARQL queries), into a binary classification task. This new dataset requires models to determine if a question and a SPARQL query have the same meaning. They employ specific strategies for generating negative examples (where the question and query don't match) to ensure the task is compositionally challenging. This includes "model negatives" derived from baseline model predictions and "random negatives." They also create two types of splits: "random split" and "maximum compound divergence (MCD) split," with MCD splits being intentionally harder for models.
  2. Structure Annotations: The paper investigates how providing structural hints to Transformer models can improve compositional generalization. They experiment with two types of structural annotations:
  3. Parse Trees: They use constituency-based parse trees for questions and dependency-based parse trees for SPARQL queries. These trees are combined to represent the input's hierarchical structure.
  4. Entity Cross-Links: These link mentions of entities (like names or concepts) that appear in both the natural language question and the SPARQL query.

    These annotations are integrated into the Transformer's attention mechanism in two ways:

  5. Hard Mask: This restricts attention to only allow connections between tokens that are directly related by the parse tree structure.

  6. Soft Mask: This assigns relative attention labels based on the structural relationships between tokens (e.g., parent-to-child, entity cross-link).

Datasets

  • CFQ (Compositional Freebase Questions) Dataset: This is the primary dataset used. The authors adapt it into a binary classification task. They create new splits ("random split" and "MCD split") and negative examples to specifically test compositional generalization.

Key Insights and Ideas

  • Structure is Key: The most significant finding is that providing explicit structural information (like parse trees) to Transformer models dramatically improves their ability to generalize compositionally. This suggests that while Transformers have some innate capacity for compositional reasoning, they need guidance to suppress non-compositional attention patterns.
  • Hard Masks are Powerful: The "hard mask" approach, which strictly limits attention based on parse tree structure, yields the most substantial gains in compositional generalization. This is particularly interesting because it doesn't add new capabilities to the model but rather removes potentially distracting attention connections.
  • Entity Links Help: Incorporating entity cross-links between the question and query also boosts performance, especially when combined with parse tree information.
  • Classification is a Good Proxy: Converting semantic parsing datasets into classification tasks is an effective way to study compositional generalization, as it simplifies evaluation while retaining the core challenge.

Key Results

  • Significant AUC Gains: In the challenging "MCD split & model negatives" dataset, models without structure annotations achieved AUC scores around 0.83-0.87. With "hard mask" and entity cross-links, this improved to 0.9656 AUC. The best result, using a combination of "soft" and "hard" masks with entity cross-links, reached 0.9721 AUC.
  • Baseline Performance: On simpler splits ("random split & random negatives"), even standard LSTM and Transformer models performed very well (AUC close to 1.0), highlighting the importance of the dataset split and negative example strategy for revealing compositional generalization deficits.
  • Hard Mask's Impact: The "hard mask" annotation alone provided substantial gains, demonstrating its effectiveness in guiding the model's attention.

Limitations and Future Work

  • Manual Annotation: The structural annotations (parse trees and entity links) were generated manually. The authors suggest that a promising future direction would be to automate this annotation process, potentially by training another model to generate these structural hints.
  • Training Time: Experiments with "hard mask" annotations required significantly more training steps, suggesting that propagating gradients through this restricted attention mechanism is slower.
  • Model Capacity vs. Guidance: The paper posits that Transformer models have the potential for compositional generalization but struggle to suppress non-compositional attention. Future work could explore how to better train models to achieve this suppression intrinsically.

Summary updated

BibTeX
@inproceedings{kim2021improving,
  author = {Juyong Kim and Pradeep Ravikumar and Joshua Ainslie and Santiago Ontañón},
  title = {{Improving Compositional Generalization in Classification Tasks via Structure Annotations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=LF2a9hoV8G},
}
Figure