TL;DR
This paper addresses the lack of publicly available commonsense knowledge bases (CSKBs) generated by COMET, a method for extracting commonsense knowledge from transformer models. They create and release four such resources, enabling a detailed analysis of COMET's strengths and weaknesses, and demonstrating their utility in downstream tasks.
What Problem Does This Paper Address?
Commonsense knowledge (CSK) is crucial for AI systems to understand and interact with the world like humans do. While various methods exist for compiling CSK, including manual efforts, text extraction, and more recently, using large language models (LLMs), a significant gap has emerged. Specifically, the COMET methodology, which fine-tunes transformer models to generate commonsense assertions, has shown great promise. However, no materialized, publicly accessible knowledge bases derived from COMET models were available. This lack of resources hinders research into the approach's effectiveness, its limitations, and its practical applications. This paper aims to fill this void by creating and analyzing these materialized CSKBs.
Methods
The authors leverage the COMET pipeline, following the implementation details from the official COMET-ATOMIC20 project. They fine-tune two language models, GPT2-XL and BART, on two established concept-centric commonsense knowledge bases: CONCEPTNET 5.7 and ASCENT++. These training resources differ in their origin: CONCEPTNET is crowd-sourced, while ASCENT++ is derived from web text extraction. This allows for an investigation into how these different training data sources impact the COMET models.
For materialization, they query the fine-tuned COMET models for 10,926 subjects from CONCEPTNET that have at least two assertions for 13 specific CSK predicates. Using beam search, they generate the top-10 completions for each subject-predicate pair, along with their beam scores. This process results in four distinct CSKBs: CONCEPTNET (GPT2-XL, BART) and ASCENT++ (GPT2-XL, BART).
Datasets
Training Datasets: * CONCEPTNET 5.7: A large, crowd-sourced commonsense knowledge graph. * ASCENT++: A resource for automated CSK extraction from web content.
Evaluation Datasets: * CSLB dataset: Used for recall evaluation, consisting of human-written sentences about property norms for 638 concepts.
The paper also discusses the original ATOMIC and ATOMIC20 resources, which were used in prior COMET work but not directly for training in this study.
Key Insights and Ideas
- COMET's potential is unlocked by materialized resources: The paper's core contribution is making COMET-generated CSK publicly available, enabling detailed analysis and practical use.
- Training data matters: The choice of training data (crowd-sourced CONCEPTNET vs. web-extracted ASCENT++) significantly impacts the performance and characteristics of the generated CSKBs.
- Recall vs. Precision Trade-off: COMET models show strong recall, often outperforming their training data, especially with CONCEPTNET. However, there's a notable drop in precision compared to the original training resources.
- LM strengths and weaknesses are evident: The generated CSKBs highlight LLMs' ability to generalize and produce novel assertions but also reveal common error patterns like co-occurrence misreadings and subject-object copying.
- Web-extracted data might be less amenable to COMET: COMET models seem to benefit more from manually curated resources like CONCEPTNET, adding less value to web-extracted data where LMs might not encounter fundamentally different text.
Key Results
- Resource Creation: Four materialized CSKBs were created, containing between 976,296 and 1,420,380 ranked assertions.
- Recall: COMET models significantly outperform manual curation in recall for CONCEPTNET and are competitive with web text extraction for ASCENT++. GPT2-XL models generally show better recall than BART models.
- Precision: A notable gap exists in precision (typicality and saliency) compared to the original training data. BART models generally demonstrate better typicality and saliency than GPT2-XL models.
- Common Errors: Identified error categories include co-occurrence misreadings, subject-object copying, quantity confusion, and redundancy.
- Downstream Use Cases: The materialized resources enable frequency aggregation, join queries, ranking, and text search, demonstrating their practical utility. For instance, the third most common location for items in GPT2-XL-CONCEPTNET was "sock drawer," a detail not prominent in the original CONCEPTNET.
Limitations and Future Work
Limitations: * Precision Gap: A significant gap in precision remains compared to manual curation, and a moderate gap exists compared to web extraction. * Error Patterns: The identified error categories (co-occurrence misreadings, subject copying, redundancy) suggest areas for improvement.
Future Work: * Post-filtering and Clustering: Further research is suggested to address common error patterns through post-filtering and clustering techniques. * Deeper Analysis: The authors plan to conduct further analyses on the strengths and weaknesses of the approach and explore more extrinsic use cases. * Resource Availability: The paper emphasizes the importance of making these resources publicly available for the advancement of the field.
Summary updated