Abstract: Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.
Introduction
Generative Commonsense Reasoning (GCR) requires models to produce coherent text that reflects an underlying understanding of physical and social realities—for instance, explaining why "a piano is not a sport." While modern Large Language Models (LLMs) have become adept at generating singular, accurate explanations, they often struggle with diversity.
In real-world scenarios, a single prompt often warrants multiple valid perspectives. Humans naturally provide varied explanations (e.g., focusing on a piano as a musical instrument, its physical keys, or its lack of athletic competition). Current AI models typically rely on "nucleus sampling" (stochastic word selection) to achieve variety. However, this often results in outputs that are syntactically different but semantically redundant, or occasionally nonsensical. The core challenge addressed by this research is the "one-to-many" generation problem: how can a model produce a set of outputs that are all valid yet vary significantly in content and perspective?
Methodology: The MoKGE Framework
The researchers introduce MoKGE (Mixture of Knowledge Graph Experts), a framework designed to bake diversity into the reasoning process itself rather than the final decoding step. The architecture consists of three primary pillars:
1. Knowledge Graph (KG) Enhancement
Rather than relying solely on the input text, MoKGE retrieves a subgraph of related concepts from ConceptNet, a massive database of commonsense knowledge. To process this structured data, the model employs a Relational Graph Convolutional Network (R-GCN), which creates "structure-aware" representations of concepts and their inter-relationships.
2. Mixture of Experts (MoE) Strategy
Instead of a monolithic decoder, MoKGE utilizes a set of specialized neural "experts." During the training phase, a stochastic hard-EM algorithm is used to encourage specialization. In this "winner-take-all" approach, only the expert that best predicts a specific reasoning path is updated for that instance. This prevents the experts from converging on a single "average" behavior, allowing one expert to focus on functional aspects of an object while another might specialize in physical properties.
3. Concept Selection and Anchoring
Each expert selects a distinct set of salient concepts from the retrieved KG subgraph. These concepts serve as "anchors" that guide a standard Transformer model. By grounding the generation in different sets of concepts, the model ensures that the resulting sentences are semantically distinct and logically sound.
Experimental Setup
The model was evaluated on two prominent GCR benchmarks:
- ComVE (CommonSense Explanation Generation): Given a counterfactual statement (e.g., "He put an elephant in the fridge"), the model must generate reasons why it is impossible.
- α-NLG (Abductive Commonsense Reasoning): Given the beginning and end of a narrative, the model must generate a plausible middle hypothesis.
Key Insights
- Reasoning-Based Diversity: The study demonstrates that diversity is most effective when it originates in the reasoning process. By forcing experts to navigate different paths through a knowledge graph, the model produces variety based on logic rather than random word choice.
- The 2-Hop Rule: Data analysis revealed that approximately 75% of concepts used by humans in these tasks are within "two hops" of the input concepts on a knowledge graph. This finding justifies the use of KGs as a bounded yet sufficiently rich search space for diverse reasoning.
- Expert Specialization: The hard-EM approach successfully created a "division of labor." Qualitative analysis showed that different experts learned to prioritize different semantic domains, such as material composition versus utility.
Results and Performance
MoKGE was benchmarked against state-of-the-art methods, including Variational Autoencoders (CVAE) and standard Nucleus Sampling.
- Human-Level Diversity: On the ComVE dataset, human annotators provide an average of 2.60 distinct meanings per prompt. MoKGE achieved a score of 2.63, slightly exceeding the human baseline in variety. In contrast, the best previous AI method (Nucleus Sampling) achieved only 2.15.
- Quality Preservation: A common pitfall in generative modeling is the "diversity-accuracy trade-off," where increased variety leads to lower accuracy scores (like BLEU or ROUGE). MoKGE maintained performance on par with top-tier models, proving that diversity does not have to come at the expense of factual correctness.
- Semantic Richness: Case studies highlighted that while baseline models often repeated the same idea with different synonyms, MoKGE provided genuinely different reasons for a situation, such as focusing on a car's fuel source in one output and its material in another.
Conclusion and Future Work
MoKGE represents a significant shift from sampling-based diversity to reasoning-based diversity. However, the authors identify several paths for future research. Currently, the model is limited by the scope of ConceptNet; future iterations could incorporate unstructured web data or retrieved documents to expand the experts' knowledge base. Additionally, the researchers call for new neural-based diversity metrics, as current automated metrics like Self-BLEU struggle to distinguish between true semantic variety and simple paraphrasing. This "one-to-many" approach holds promise for broader applications, including open-ended dialogue and complex story generation.
Summary updated