TL;DR
This paper tackles the brittleness of powerful language models like BERT on Natural Language Inference (NLI) tasks when faced with syntactic variations. They propose a novel approach that integrates dependency syntax into BERT using a graph convolutional network (GCN) to create a richer, syntax-aware representation, significantly improving generalization on challenging NLI examples.
What Problem Does This Paper Address?
Natural Language Inference (NLI) is a fundamental task in NLP, aiming to determine the relationship between two sentences (entailment, contradiction, or neutral). While large pre-trained models like BERT have achieved state-of-the-art performance, they often struggle with generalization. Specifically, models can be easily fooled by superficial syntactic changes, even when the core meaning remains similar or the lexical overlap is high. This vulnerability is highlighted by datasets like HANS (Heuristic Analysis for NLI Systems), which contain examples designed to expose these weaknesses. The paper aims to make NLI models, particularly BERT, more robust to these syntactic variations and improve their ability to generalize to unseen data with different sentence structures.
Methods
The core of the proposed method involves augmenting BERT's representations with syntactic information using a Graph Convolutional Network (GCN). Here's a breakdown:
- Dependency Parsing: For each sentence pair, dependency trees are generated using a neural parser.
- Graph Construction: These dependency trees are then used to construct a unified graph representing the sentence pair. This graph includes:
- Dependency Edges: Representing the syntactic relationships within each sentence.
- Reverse Dependency Edges: Allowing information flow against the syntactic direction.
- Self-Loops: To preserve individual word information.
- Co-Attention Edges: A novel addition that explicitly links words between the premise and hypothesis based on their similarity. This creates a "soft co-attention matching pattern" that integrates the two sentences.
- Graph Convolutional Network (GCN): A GCN is employed to process this integrated graph. The GCN iteratively updates node (word) representations by aggregating information from their neighbors, effectively propagating syntactic and cross-sentence contextual information. Highway units are used to help preserve information across multiple GCN layers.
- Co-Attention Layer: After the GCN layers, a co-attention layer further refines the representations by explicitly modeling the interaction between the premise and hypothesis representations.
- Classifier: The final representations are concatenated with their element-wise difference and multiplication and fed into a linear layer with a softmax activation for classification.
The key novelty lies in constructing a single, integrated graph for the sentence pair that incorporates both intra-sentence syntactic dependencies and inter-sentence co-attention, and then processing this graph with a GCN. This differs from prior work that might encode sentences separately or only use syntax as an input feature to a classifier.
Datasets
- Training: The models are trained on the MNLI (Multi-Genre Natural Language Inference) dataset.
- Evaluation: Evaluation is performed on:
- MNLI: This dataset includes both in-domain (MNLI-m) and cross-domain (MNLI-mm) splits.
- HANS (Heuristic Analysis for NLI Systems): This dataset is specifically designed to test generalization by including examples that exploit common NLI heuristics, particularly focusing on "non-entailed lexical overlap" categories where premise and hypothesis share many words but have different syntactic structures.
Key Insights and Ideas
- Syntax Matters for Generalization: The paper strongly supports the idea that explicit syntactic information is crucial for NLI models to generalize beyond the specific patterns learned during training.
- Integrated Graph Representation is Powerful: Creating a single graph that captures both intra-sentence syntax and inter-sentence word alignment (via co-attention) allows for richer contextualization than processing sentences independently.
- Co-Attention Links are Key for Cross-Sentence Syntax: The proposed co-attention edges between words in the premise and hypothesis are particularly effective in bridging syntactic differences and improving performance on challenging HANS examples.
- BERT's Brittleness: The results on HANS clearly demonstrate that even powerful models like BERT can be easily misled by syntactic variations, highlighting the need for more robust methods.
Key Results
- The proposed BERT+CAGCN model significantly outperforms BERT baselines on the "non-entailed lexical overlap" category of HANS, achieving much higher accuracy. For instance, on this specific category, BERT+CAGCN achieves 63.5% accuracy, while BERT-CLS achieves only 57.9%.
- The model shows improved performance on the overall six categories of HANS, demonstrating better generalization.
- Performance on the MNLI dataset (both in-domain and cross-domain) is slightly improved or maintained, indicating that the syntax integration does not degrade performance on standard NLI tasks.
- Comparing different GCN linking strategies, the co-attention links (BERT+CAGCN) yield better results than simply linking same lemmas (BERT+SWGCN) or no links (BERT+SGCN).
Limitations and Future Work
- Passive Subcategory: The proposed GCN-based methods did not show significant improvement on the "Passive" subcategory of HANS, nor on "Non-entailment-subsequence" and "Non-entailment-constituent" categories. The authors suggest this might be because these categories often have similar syntactic structures and word positions, making it harder for syntax to provide a distinct advantage.
- MNLI Bias: The paper notes that MNLI has certain biases (e.g., more supporting labels, negation patterns) that might influence model learning. While HANS was created to address this, further work could explore how to mitigate these inherent dataset biases more effectively.
- Syntactic Feature as Secondary Factor: For MNLI, the syntactic feature might be learned as a secondary factor compared to content and negation, explaining only slight improvements on this dataset. Future work could explore how to make syntax a more primary signal.
- Future Directions: The authors suggest that incorporating inductive biases like dependency trees via GCN can make sentence encoding more robust, implying that exploring other forms of structural information or more sophisticated graph representations could be fruitful.
Summary updated