TL;DR
This paper investigates whether machines can learn fundamental "common sense" distinctions (like class vs. instance, or physical object vs. not) directly from Linked Open Data (LOD). They developed and evaluated machine learning and alignment-based methods, finding that LOD implicitly encodes these distinctions and that machine learning approaches, particularly Support Vector Machines, achieve strong results, suggesting LOD is a valuable resource for common sense knowledge.
What Problem Does This Paper Address?
The Semantic Web, particularly Linked Open Data (LOD), aims to make global knowledge machine-readable. However, much of LOD lacks the nuanced, foundational distinctions that humans use effortlessly – for example, knowing if "dog" refers to the concept of a dog (a class) or a specific dog (an instance), or if something is a physical object. These distinctions are crucial for many AI tasks, like natural language understanding. While foundational ontologies (like DOLCE and SUMO) formalize these concepts, they are often developed top-down and not well-integrated with the bottom-up, data-driven nature of LOD. This paper bridges this gap by exploring if machines can learn these common sense distinctions directly from LOD, and if these learned distinctions align with human intuition.
Methods
The authors employ two main approaches:
- Alignment-based Classification (SENECA): This method leverages existing links and alignments within LOD. SENECA uses these connections to infer whether an entity is a class or an instance (by checking links to resources like WordNet or Wiktionary) or a physical object (by navigating taxonomies like YAGO and OntoWordNet). This approach is unsupervised.
- Machine Learning-based Classification: This involves training various classification algorithms (J48, Random Forest, SVM, etc.) on features extracted from LOD entities. The key features used are:
- Abstract: The textual description of the entity, treated as a bag-of-words.
- URI: Features derived from the entity's Uniform Resource Identifier, such as the number of capitalized terms.
- Incoming/Outgoing Properties: The types of relationships an entity participates in, as subject or object in RDF triples.
- SENECA Output: The result from the alignment-based method, used as an additional feature.
They also explore weighting training data based on crowd agreement, allowing models to learn from ambiguity.
Datasets
The paper introduces four new, reusable datasets for evaluating the learned distinctions:
- CIE (Class vs. Instance Expert Annotated): Manually annotated by two experts.
- CIC (Class vs. Instance Crowd Annotated): Annotated by crowd workers, with agreement scores calculated.
- POE (Physical Object Expert Annotated): Manually annotated by two experts.
- POC (Physical Object Crowd Annotated): Annotated by crowd workers, with agreement scores calculated.
These datasets are based on a sample of DBpedia entities and are designed to assess whether the learned distinctions align with common sense.
Key Insights and Ideas
- LOD as a Source of Common Sense: The core idea is that LOD, despite its encyclopedic nature, implicitly encodes common sense distinctions. The high agreement between crowd annotations and expert judgments strongly supports this.
- Machine Learning Outperforms Alignment: While SENECA provides a useful baseline and feature, machine learning models, particularly Support Vector Machines, achieve superior performance, especially when trained on a diverse set of features.
- Feature Importance: The entity's abstract (textual description) and its incoming/outgoing properties (relational context) are consistently strong predictors for both distinctions. The URI structure is particularly informative for the class vs. instance distinction.
- Crowdsourcing as a Validation Tool: The use of crowdsourcing not only generates large-scale annotations but also provides a measure of "common sense" by comparing crowd judgments against expert ones. High agreement suggests the distinctions are indeed intuitive.
- Ambiguity is Informative: Weighting training data by crowd agreement allows machine learning models to learn from ambiguous cases, potentially leading to a richer understanding of concepts.
Key Results
- Class vs. Instance:
- SENECA achieved an average F1 score of .836 when compared against the crowd-annotated CIC dataset.
- Support Vector Machines, using a combination of all features (Abstract, URI, Properties, SENECA output) and trained on crowd-annotated data (CIC) with agreement > 80%, achieved an average F1 score of .941.
- Physical Object vs. Not Physical Object:
- SENECA's performance dropped significantly, with an average F1 score of .557, possibly due to more complex alignment paths.
- Support Vector Machines, using a combination of Abstract, Properties, and SENECA output, achieved an average F1 score of .868 when trained on crowd-annotated data (POC) with agreement > 80%.
- Common Sense Alignment: The agreement between crowd annotations and expert annotations for both distinctions was very high (e.g., 95.7% for class/instance, 85.69% for physical object), indicating that the learned distinctions align well with common sense.
Limitations and Future Work
- Generalizability: While the methods show promise, the authors acknowledge that the results might be influenced by the "standardized" style of writing in DBpedia abstracts and infoboxes. Further validation on more diverse LOD sources is needed.
- Feature Engineering: The URI feature was more effective for class/instance than for physical object classification, suggesting that feature sets might need to be tailored to specific distinctions.
- Future Directions:
- Expanding the study to other foundational distinctions (e.g., physical vs. social objects, types of locations).
- Investigating more sophisticated features derived from LOD.
- Analyzing LOD at a much larger scale using tools like LOD Laundromat.
- Developing methods to learn relational knowledge (frames) from LOD.
- Exploring the distinction between physical objects that can serve as locations or containers.
Summary updated