TL;DR
This paper introduces KGSS, a novel unsupervised method for generating knowledge graphs (KGs) from text. KGSS leverages semantic similarity to automatically identify relationships between entities, eliminating the need for labeled data or manual mapping to predefined schemas. It significantly outperforms existing unsupervised KG generation systems across multiple datasets.
What Problem Does This Paper Address?
Knowledge Graphs (KGs) are powerful structured representations of information, but their creation is challenging. Traditional methods often rely on supervised or semi-supervised approaches, which require extensive labeled data that is expensive and time-consuming to create. Unsupervised methods exist, but they tend to generate too many redundant or inaccurate relations, or they still require manual effort to map extracted relations to a known schema. This paper addresses the need for a truly unsupervised KG generation system that is both accurate and flexible, capable of working with any desired relation schema without requiring manual annotation.
Methods
The core of KGSS lies in its unsupervised approach to relation extraction, driven by semantic similarity. The process involves:
- Entity Extraction: Identifying named entities (like persons, organizations, locations) and optionally noun phrases within sentences. This step utilizes the spaCy library for efficiency.
- Entity Tuple Formation and Filtering: Creating pairs of entities (head-tail) from each sentence and applying heuristic rules to filter out unlikely pairs based on entity types (e.g., a location is unlikely to have a "spouse" relation with a person).
- Relation Extraction: This is the key innovation. For each entity pair and its surrounding text (Psub), KGSS uses a pre-trained language model (Sentence-BERT) to compute semantic similarity between Psub and candidate relations (Ri). The relation with the highest similarity score above a certain threshold is selected. This allows for flexible relation extraction without a predefined set.
- KG Storage and Visualization: The generated triples are stored and can be visualized.
The system also offers optional pattern-based rules to further refine relation extraction in specific domains like news articles, improving performance when entities are adjacent in text.
Datasets
The paper evaluates KGSS on four datasets:
- TACRED*: A subset of the TACRED dataset, excluding "no_relation" instances.
- NYT: A dataset derived from New York Times articles.
- WEBNLG: A dataset for natural language generation.
- NewsKG21: A newly created dataset by the authors, consisting of 685 sentences from recent news articles (CNN, CBC, USNEWS, The Star, Wikipedia News) with 1247 manually generated triples. This dataset was created to address potential quality issues in existing benchmark KGs and to provide a benchmark specifically for the news domain.
Key Insights and Ideas
- Semantic Similarity as the Core: The paper demonstrates that leveraging semantic similarity from pre-trained language models (like Sentence-BERT) is a powerful and flexible way to identify relations between entities in an unsupervised manner.
- Flexibility in Relation Schemas: KGSS is designed to work with any set of relations a user prefers, a significant advantage over supervised methods that are tied to their training data's schema.
- Effective Entity Type Filtering: The heuristic rules for filtering entity tuples based on entity types (e.g., only allowing "person" type entities for "spouse" relations) significantly improve the quality of extracted triples by removing noisy pairs.
- Unsupervised Methods Can Compete: The results show that KGSS, a purely unsupervised method, can outperform even sophisticated baselines, highlighting the potential of unsupervised learning for KG generation.
- Importance of Domain-Specific Rules: The optional pattern-based rules show that incorporating domain knowledge can further boost performance, especially in scenarios where standard semantic similarity might struggle with adjacent entities.
Key Results
- Superior Performance: KGSS consistently outperforms two state-of-the-art unsupervised baselines (Stanford OpenIE and MAMA) across all evaluated datasets (TACRED*, NYT, WEBNLG, NewsKG21) on KG triple extraction, achieving considerably higher precision, recall, and F1 scores. For instance, on the NewsKG21 dataset, KGSS achieved an F1 score of 22.3%, compared to 8.7% for Stanford OpenIE and 3.2% for MAMA.
- Entity Tuple Extraction: KGSS also shows strong performance in entity tuple extraction, outperforming baselines on the NewsKG21 dataset.
- Impact of Rules: Applying relation extraction rules (RE1, RE2, RE3) and entity tuple filtering rules (TF1, TF2, TF3) consistently improves KGSS's performance, with the combined rules leading to a significant boost.
- Human Evaluation Confirmation: Human evaluation on the NewsKG21 dataset also confirmed KGSS's superiority over Stanford OpenIE and MAMA, indicating that its performance is not just an artifact of automatic metrics.
Limitations and Future Work
- Dependency on NER Tools: The effectiveness of KGSS is partly dependent on the accuracy of the underlying Named Entity Recognition (NER) tools.
- Relation Schema Similarity: The performance of the semantic matching phase might be less effective when relation names in the schema are very similar.
- Supervised vs. Unsupervised: While KGSS excels in unsupervised settings, supervised methods can achieve better results if training data is available.
- Future Work: The authors plan to evaluate the effectiveness of using their method for bootstrapping knowledge graph generation with distant supervision. They also developed an open-source interactive KG generation and visualization tool.
Summary updated