TL;DR
This paper introduces a novel framework for zero-shot commonsense question answering that tackles the limitation of previous methods by incorporating multi-hop reasoning from knowledge graphs. By generating synthetic multi-hop QA data and using contrastive learning for pre-training, their approach significantly improves performance on various commonsense QA benchmarks, demonstrating a more robust understanding of nuanced reasoning.
What Problem Does This Paper Address?
Commonsense question answering (QA) is crucial for machines to understand and interact with the world like humans. While large language models have made strides, they often struggle with nuanced reasoning that requires connecting multiple pieces of information. Existing zero-shot QA methods typically rely on single-hop relationships from knowledge graphs (KGs) to create training data. This approach overlooks the fact that many commonsense questions necessitate multi-hop reasoning – connecting several pieces of information in a logical sequence. For instance, understanding why a revolving door might be a security measure at a bank requires linking "revolving door" to "at location" to "bank," and then potentially to "security." Previous methods, by only using single-hop connections, fail to capture this deeper reasoning, leading to incorrect answers, especially when faced with confusing distractors. This paper aims to bridge this gap by developing a framework that explicitly injects multi-hop commonsense knowledge into models for zero-shot QA.
Methods
The proposed framework consists of two main stages: synthetic multi-hop QA generation and knowledge injection pre-training.
- Synthetic Multi-hop QA Generation: This stage focuses on creating realistic QA pairs that capture multi-hop reasoning. The authors propose two novel methods:
- Compositive Commonsense QA Generation: This method extracts multi-hop relationships that represent continuous reasoning, akin to a chain of thought. For example, "university" -> "library" -> "books." It involves finding two related triples where the tail of one is the head of another, masking the common entity, and using it as the question and correct answer.
-
Conjunctive Commonsense QA Generation: This method captures multi-faceted reasoning, where a single entity has multiple attributes or uses. For instance, a "gym" can be associated with "basketball," "football," etc. It involves finding triples with the same head entity but different relations/tails, masking the head entity, and using it as the question and answer. Both methods are designed to generate "hard negative samples" – distractors that are semantically related but incorrect – to improve the model's discriminative power.
-
Knowledge Injection Pre-training: After generating the synthetic multi-hop QA dataset (and optionally combining it with single-hop data), the model is pre-trained using contrastive learning. Specifically, they concatenate the question with each answer option (positive for the correct answer, negative for distractors) to form input sequences. These sequences are then fed into a masked language model (like RoBERTa). The model learns to assign a higher score (lower MLM loss) to the positive QA pair and lower scores to negative pairs. This contrastive objective, using the InfoNCE loss, encourages the model to internalize the multi-hop commonsense knowledge embedded in the synthetic data.
Datasets
The paper evaluates its framework on five standard commonsense question answering benchmarks:
- CommonsenseQA (CSQA)
- Multiple Choice Temporal commonsense (MC-TACO)
- Abductive NLI (a-NLI)
- PhysicalIQA (PIQA)
- Wino-Grande (WG)
For generating synthetic data, the authors primarily use ConceptNet, chosen for its rich multi-hop relationships. They also mention using ATOMIC, WikiData, and WordNet in prior work and for single-hop data generation.
Key Insights and Ideas
- The critical role of multi-hop reasoning: The paper strongly emphasizes that many commonsense questions cannot be answered correctly with only single-hop knowledge. The example of the revolving door highlights how connecting multiple facts is essential for accurate reasoning.
- Generating "hard" negatives is crucial: Simply picking random distractors isn't enough. The proposed methods for generating semantically similar but incorrect answers significantly boost performance, indicating that models need to learn to distinguish subtle differences.
- Contrastive learning is effective for knowledge injection: By framing the problem as distinguishing correct from incorrect multi-hop reasoning paths, contrastive learning provides a powerful mechanism to embed this knowledge into the model during pre-training.
- Linguistic logic in KG reasoning: The authors' focus on extracting multi-hop paths that conform to linguistic logic (compositive and conjunctive) suggests a more principled way to leverage KGs beyond simple triple extraction.
Key Results
The proposed "Multi-hop Knowledge Inject Framework" achieved state-of-the-art performance on all five evaluated commonsense QA benchmarks in a zero-shot setting.
- It significantly improved accuracy, for example, by ~2.8% on CSQA and ~5.3% on MC-TACO compared to previous baselines.
- Ablation studies confirmed the effectiveness of each component: removing hard negative answers degraded performance by 1.3%, and removing all multi-hop knowledge led to a significant drop, underscoring the framework's contribution.
- The framework also demonstrated better performance than previous methods like RoBERTa-L (MR) and Zero-shot fusion (Adapter), while being more computationally efficient (fewer parameters, fewer epochs) than some adapter-based approaches.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Limited KG scope: The current framework primarily utilizes ConceptNet for multi-hop reasoning. Other KGs like ATOMIC have less explicit multi-hop structures, and exploring how to extract implicit multi-hop relations from them is a key future direction.
- Focus on two hops: The current generation methods consider entities and relationships within two hops. Future work could explore methods to utilize nodes beyond two hops to enrich multi-hop knowledge further.
- Low-resource efficiency: While the framework is more efficient than some adapter-based methods, further research into low-resource settings could be beneficial.
- Generalizability: While the framework shows strong results, exploring its performance on a wider variety of commonsense reasoning tasks and datasets would be valuable.
Summary updated