Abstract
This paper introduces the Multi-Head Knowledge Attention (MHKA) model, designed to bridge the gap in social commonsense reasoning for large language models. By dynamically generating and attending to semi-structured "if-then" social rules (e.g., mental states and social consequences), the model outperforms strong baselines like RoBERTa on abductive reasoning and a newly introduced task: Counterfactual Invariance Prediction.
Introduction: The Challenge of Social Commonsense
While Pre-trained Language Models (PLMs) are excellent at capturing linguistic patterns, they often struggle with "social commonsense"—the ability to infer the unstated intentions, emotions, and social norms behind human behavior.
The authors identify two main bottlenecks in current approaches: Knowledge Acquisition (obtaining relevant social knowledge not explicitly in the text) and Knowledge Integration (effectively injecting this knowledge into a neural architecture so the model reasons with it rather than treating it as noise).
Furthermore, the paper explores the link between abductive reasoning (finding the most plausible explanation for an event) and counterfactual reasoning (predicting if an outcome would stay the same if a previous event were different).
Methodology: Multi-Head Knowledge Attention (MHKA)
The authors propose the MHKA model, which consists of three main modules:
- Context Encoding Layer: Uses a standard PLM (like RoBERTa) to create representations of the input text.
- Knowledge Encoding Layer: Instead of using a static knowledge graph, the authors use COMET (a GPT-2 based model trained on the ATOMIC dataset) to dynamically generate contextually relevant social rules covering dimensions like intent, need, and reaction.
- Reasoning Cell: This core innovation uses a multi-head attention mechanism to allow the model to jointly attend to the input text and multiple pieces of generated knowledge. This allows the model to "emulate" reasoning steps by recursively refining the representation based on different social rules.
To prepare the data, the system uses Semantic Role Labeling (SRL) to identify "who did what to whom," ensuring the generated knowledge is correctly mapped to the characters in the story.
Datasets and Evaluation
The model was evaluated on three primary benchmarks:
- $\alpha$NLI (Abductive Natural Language Inference): A dataset where the model must choose the most plausible explanation ($H_1$ or $H_2$) to bridge two observations ($O_1$ and $O_2$).
- CIP (Counterfactual Invariance Prediction): A newly introduced task where, given a story, if an intermediate event is replaced by a counterfactual one, the model must predict if the final outcome remains "invariant" (unchanged).
- ATOMIC: Used via COMET to generate the underlying social commonsense rules.
Key Insights and Results
- Counterfactuals Aid Abduction: A significant finding is that training to understand counterfactuals improves abductive reasoning. Knowing what wouldn't happen if circumstances changed helps the model identify the most likely cause of what did happen.
- State-of-the-Art Performance: On the $\alpha$NLI task, RoBERTa-L + MHKA achieved 87.12% accuracy, a significant improvement over the RoBERTa-L baseline (84.48%).
- Low-Resource Robustness: In scenarios with very little training data (e.g., 1% or 2%), the MHKA model showed even more substantial gains, proving that explicit knowledge helps when data is scarce.
- Human Validation: Human evaluators confirmed that the model's attention was often focused on the most relevant knowledge rules (approx. 68% of the time), suggesting the model is actually utilizing the provided commonsense.
Conclusion and Future Work
While MHKA effectively integrates social knowledge, it is limited by the quality of the knowledge COMET generates. Future directions include investigating more complex compositional effects—how multiple social rules interact and conflict in more intricate narratives—and reducing the computational overhead of the additional encoding layers.
Summary updated