Proceedings of the 2nd Workshop on Deep Learning on Graphs for Natural Language Processing (DLG4NLP 2022)

Wenhao Yu, Chenguang Zhu, Lianhui Qin, Zhihan Zhang, Tong Zhao, Meng Jiang, Jin Guo, Zhen Han, Zhou Su, Jiliang Li, Volker Tresp, Yuyi Wang, Suk Woo, Yu-Jung Choi, Dharani Heo, Byoung-Tak Punithan, Zhenyun Deng, Yonghua Zhu, Qianqian Qi, Michael Witbrock, Patricia Riddle, Mérième Bouhandi, Emmanuel Morin, Thierry Hamon, Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Sasanka Ammanamanchi, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Changmao Li, Jeffrey Flanigan, Donghan Yu, Yuwei Fang, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, Michael Zeng, Zaitang Li, Zhiting Hu, Qingyun Wang, Ji Heng, Zhichun Guo, Qingkai Zeng, Jinfeng Lin, Jane Cleland-Huang, Houyu Zhang, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, Yoav Artzi, Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, Bill Dolan, Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, Xiaoyan Zhu, Yaoming Zhu, Sidi Lu, ;Openalex Unknown 2022 · DOI · OpenAlex

Abstract: Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.

Introduction

Generative Commonsense Reasoning (GCR) requires models to produce coherent text that reflects an underlying understanding of physical and social realities—for instance, explaining why "a piano is not a sport." While modern Large Language Models (LLMs) have become adept at generating singular, accurate explanations, they often struggle with diversity.

In real-world scenarios, a single prompt often warrants multiple valid perspectives. Humans naturally provide varied explanations (e.g., focusing on a piano as a musical instrument, its physical keys, or its lack of athletic competition). Current AI models typically rely on "nucleus sampling" (stochastic word selection) to achieve variety. However, this often results in outputs that are syntactically different but semantically redundant, or occasionally nonsensical. The core challenge addressed by this research is the "one-to-many" generation problem: how can a model produce a set of outputs that are all valid yet vary significantly in content and perspective?

Methodology: The MoKGE Framework

The researchers introduce MoKGE (Mixture of Knowledge Graph Experts), a framework designed to bake diversity into the reasoning process itself rather than the final decoding step. The architecture consists of three primary pillars:

1. Knowledge Graph (KG) Enhancement

Rather than relying solely on the input text, MoKGE retrieves a subgraph of related concepts from ConceptNet, a massive database of commonsense knowledge. To process this structured data, the model employs a Relational Graph Convolutional Network (R-GCN), which creates "structure-aware" representations of concepts and their inter-relationships.

2. Mixture of Experts (MoE) Strategy

Instead of a monolithic decoder, MoKGE utilizes a set of specialized neural "experts." During the training phase, a stochastic hard-EM algorithm is used to encourage specialization. In this "winner-take-all" approach, only the expert that best predicts a specific reasoning path is updated for that instance. This prevents the experts from converging on a single "average" behavior, allowing one expert to focus on functional aspects of an object while another might specialize in physical properties.

3. Concept Selection and Anchoring

Each expert selects a distinct set of salient concepts from the retrieved KG subgraph. These concepts serve as "anchors" that guide a standard Transformer model. By grounding the generation in different sets of concepts, the model ensures that the resulting sentences are semantically distinct and logically sound.

Experimental Setup

The model was evaluated on two prominent GCR benchmarks:

  • ComVE (CommonSense Explanation Generation): Given a counterfactual statement (e.g., "He put an elephant in the fridge"), the model must generate reasons why it is impossible.
  • α-NLG (Abductive Commonsense Reasoning): Given the beginning and end of a narrative, the model must generate a plausible middle hypothesis.

Key Insights

  • Reasoning-Based Diversity: The study demonstrates that diversity is most effective when it originates in the reasoning process. By forcing experts to navigate different paths through a knowledge graph, the model produces variety based on logic rather than random word choice.
  • The 2-Hop Rule: Data analysis revealed that approximately 75% of concepts used by humans in these tasks are within "two hops" of the input concepts on a knowledge graph. This finding justifies the use of KGs as a bounded yet sufficiently rich search space for diverse reasoning.
  • Expert Specialization: The hard-EM approach successfully created a "division of labor." Qualitative analysis showed that different experts learned to prioritize different semantic domains, such as material composition versus utility.

Results and Performance

MoKGE was benchmarked against state-of-the-art methods, including Variational Autoencoders (CVAE) and standard Nucleus Sampling.

  • Human-Level Diversity: On the ComVE dataset, human annotators provide an average of 2.60 distinct meanings per prompt. MoKGE achieved a score of 2.63, slightly exceeding the human baseline in variety. In contrast, the best previous AI method (Nucleus Sampling) achieved only 2.15.
  • Quality Preservation: A common pitfall in generative modeling is the "diversity-accuracy trade-off," where increased variety leads to lower accuracy scores (like BLEU or ROUGE). MoKGE maintained performance on par with top-tier models, proving that diversity does not have to come at the expense of factual correctness.
  • Semantic Richness: Case studies highlighted that while baseline models often repeated the same idea with different synonyms, MoKGE provided genuinely different reasons for a situation, such as focusing on a car's fuel source in one output and its material in another.

Conclusion and Future Work

MoKGE represents a significant shift from sampling-based diversity to reasoning-based diversity. However, the authors identify several paths for future research. Currently, the model is limited by the scope of ConceptNet; future iterations could incorporate unstructured web data or retrieved documents to expand the experts' knowledge base. Additionally, the researchers call for new neural-based diversity metrics, as current automated metrics like Self-BLEU struggle to distinguish between true semantic variety and simple paraphrasing. This "one-to-many" approach holds promise for broader applications, including open-ended dialogue and complex story generation.

Summary updated

Abstract

Generative commonsense reasoning (GCR) in natural language is to reason about the commonsense while generating coherent text. Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. Nevertheless, these approaches have seldom investigated diversity in the GCR tasks, which aims to generate alternative explanations for a real-world situation or predict all possible outcomes. Diversifying GCR is challenging as it expects to generate multiple outputs that are not only semantically different but also grounded in commonsense knowledge. In this paper, we propose MoKGE, a novel method that diversifies the generative reasoning by a mixture of expert (MoE) strategy on commonsense knowledge graphs (KG). A set of knowledge experts seek diverse reasoning on KG to encourage various generation outputs. Empirical experiments demonstrated that MoKGE can significantly improve the diversity while achieving on par performance on accuracy on two GCR benchmarks, based on both automatic and human evaluations.
BibTeX
@inproceedings{yu2022proceedings,
  author = {Wenhao Yu and Chenguang Zhu and Lianhui Qin and Zhihan Zhang and Tong Zhao and Meng Jiang and Jin Guo and Zhen Han and Zhou Su and Jiliang Li and Volker Tresp and Yuyi Wang and Suk Woo and Yu-Jung Choi and Dharani Heo and Byoung-Tak Punithan and Zhenyun Deng and Yonghua Zhu and Qianqian Qi and Michael Witbrock and Patricia Riddle and Mérième Bouhandi and Emmanuel Morin and Thierry Hamon and Chandra Bhagavatula and Ronan Le Bras and Chaitanya Malaviya and Sebastian Gehrmann and Tosin Adewumi and Karmanya Aggarwal and Sasanka Ammanamanchi and Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Changmao Li and Jeffrey Flanigan and Zhou Su and Changmao Li and Jeffrey Flanigan and Zhou Su and Donghan Yu and Chenguang Zhu and Yuwei Fang and Wenhao Yu and Shuohang Wang and Yichong Xu and Xiang Ren and Yiming Yang and Michael Zeng and Wenhao Yu and Chenguang Zhu and Yuwei Fang and Donghan Yu and Shuohang Wang and Yichong Xu and Michael Zeng and Meng Jiang and Wenhao Yu and Chenguang Zhu and Zaitang Li and Zhiting Hu and Qingyun Wang and Ji Heng and Meng Jiang and Wenhao Yu and Chenguang Zhu and Tong Zhao and Zhichun Guo and Meng Jiang and Qingkai Zeng and Jinfeng Lin and Wenhao Yu and Jane Cleland-Huang and Meng Jiang and Houyu Zhang and Zhenghao Liu and Chenyan Xiong and Zhiyuan Liu and Tianyi Zhang and Varsha Kishore and Felix Wu and Kilian Weinberger and Yoav Artzi and Yizhe Zhang and Michel Galley and Jianfeng Gao and Zhe Gan and Xiujun Li and Chris Brockett and Bill Dolan and Hao Zhou and Tom Young and Minlie Huang and Haizhou Zhao and Jingfang Xu and Xiaoyan Zhu and Yaoming Zhu and Sidi Lu and Michael J. Witbrock and Patricia J. Riddle and Tosin P. Adewumi and ;},
  title = {{Proceedings of the 2nd Workshop on Deep Learning on Graphs for Natural Language Processing (DLG4NLP 2022)}},
  booktitle = {Openalex Unknown},
  year = {2022},
  doi = {10.18653/v1/2022.dlg4nlp-1},
}
Figure