TL;DR
This paper investigates how to improve the ability of large language models (LLMs) to perform compositional generalization on the task of kinship prediction from stories. They found that data augmentation significantly boosts performance, while using predicted intermediate kinship graphs surprisingly degrades it.
What Problem Does This Paper Address?
Transformer-based LLMs have achieved impressive results across many Natural Language Processing (NLP) tasks. However, a key limitation is their struggle with compositional generalization. This means that while they can understand and reason about concepts they've seen, they often fail when these concepts are combined in novel ways. For instance, if a model knows "father" and "sister," it might struggle to infer the relationship between "uncle" and "nephew" if it hasn't seen that specific composition. This paper focuses on this challenge within the specific task of kinship prediction from narrative text, where inferring family relationships often requires combining multiple pieces of information.
Methods
The authors propose and evaluate two strategies to enhance compositional generalization for kinship prediction:
- Data Augmentation: This strategy involves generating additional training examples by identifying entity pairs within a story whose relationships are explicitly stated. For example, if a story mentions "Jenny and her father Rob," and the query is about Jenny and Rob, this is a direct relationship. The augmentation process creates more such direct examples to help the model better learn to extract explicit relations, which is a prerequisite for more complex compositional reasoning.
- Intermediate Kinship Graphs: This approach aims to improve reasoning by first predicting a structured representation of the kinship relationships within a story (a kinship graph) and then jointly reasoning over this graph and the original text. The idea is that the explicit structure of the graph might guide the model's inference.
The base model used is an adaptation of the RoBERTa-based architecture, with finetuning of the transformer layers. For the kinship graph approach, they also incorporate Graph Attention Networks (GATs) or Relational Graph Convolutional Networks (R-GCNs) to process the graph structure.
Datasets
The primary dataset used for training and evaluation is CLUTRR (Sinha et al., 2019). CLUTRR is a benchmark designed to test inductive reasoning from text, specifically focusing on kinship prediction. Each example in CLUTRR consists of a story (S), a ground-truth kinship graph (G), and a pair of query entities (e1, e2). The dataset is structured such that test sets are categorized by 'k', the number of composition operations required to infer the relationship between the query entities, ranging from 2 to 10. The authors also explore scenarios with "noisy" training and test sets, categorized by supporting facts, irrelevant facts, and disconnected facts, as defined in prior work.
Key Insights and Ideas
- Data augmentation is a powerful tool for compositional generalization: The paper strongly supports the idea that augmenting training data with explicitly stated relationships significantly improves a model's ability to generalize to novel compositions. This suggests that improving the foundational skill of relation extraction is crucial for higher-level reasoning.
- Intermediate structured representations can be detrimental: Counter-intuitively, predicting and using intermediate kinship graphs degraded the model's generalization performance. This suggests that the proposed method of fusing text and graph representations might be suboptimal, or that the errors introduced by predicting the graph outweigh its benefits.
- Robustness to noisy data: The models, particularly the base and graph-augmented versions, showed a tendency to perform better on noisy test sets (with supporting, irrelevant, or disconnected facts) compared to clean sets requiring the same number of composition steps. This indicates that the models are adept at identifying useful information within the graph structure.
- Data augmentation is effective even with limited supervision: The benefits of data augmentation persist even when only a small percentage (1% or 10%) of the training data is augmented, highlighting its efficiency.
Key Results
- Data augmentation boosted generalization performance by approximately 20% on average compared to a baseline model without these strategies.
- Using intermediate kinship graphs led to a deterioration in generalization performance by around 50% on average, relative to models that only leveraged data augmentation.
- The data augmentation strategy showed improvements even when tested on relations requiring up to k=10 composition steps, demonstrating its effectiveness for complex reasoning.
- In low-data regimes, data augmentation continued to provide performance boosts, even when the training dataset size was reduced by an order of magnitude.
Limitations and Future Work
- Sub-optimal fusion of modalities: The authors hypothesize that the method of fusing text and graph representations in the "SSD (graph)" model might be sub-optimal, leading to the observed performance degradation. They suggest that future work should focus on devising better techniques for this fusion.
- Feasibility of gold kinship graphs: While gold kinship graphs were used to generate augmented data and intermediate graphs, the authors acknowledge that obtaining such gold graphs might not be feasible in real-world scenarios. They explored scenarios with limited additional annotation (1% and 10%) and found promising results, suggesting this is a viable direction.
- Exploring alternative graph reasoning methods: The paper's findings on intermediate kinship graphs suggest that the current approach to graph reasoning might need refinement. Future work could explore different graph neural network architectures or fusion strategies.
Summary updated