Towards Comprehensive Noise Detection in Automatically Created Knowledge Graphs

Nandana Mihindukulasooriya, Oktie Hassanzadeh, Sarthak Dash, Alfio GliozzoISWC (Posters, Demos & Industry Tracks) 2017 · OpenReview

Introduction

Knowledge Graphs (KGs) serve as the backbone for modern AI applications, but their construction at scale often relies on automatic information extraction from massive, unstructured text corpora. This process is inherently noisy. While most KG refinement efforts focus on binary "fact-checking," this paper argues that noise is not a monolith. A triple can be problematic not just because it is factually false, but because it is logically inconsistent or overly vague ("generic").

Noise Taxonomy

A key contribution of this research is the formal classification of noise into distinct categories: Inconsistent, Generic, and Factual. By distinguishing between these types, systems can apply specialized filtering techniques rather than a one-size-fits-all model. This taxonomy allows for more efficient and nuanced data cleaning.

Taxonomy of noise types in extracted triples

Methods: The Sequential Noise Detection Workflow

The authors propose a multi-stage pipeline that acts as a funnel, filtering out noise in order of computational complexity and logical priority:

  1. Inconsistent Triple Filter: This stage identifies triples that violate ontological constraints. By mapping entities and relations to high-quality external KGs like DBpedia, the system checks for domain and range violations (e.g., a "sibling of" relation where the object is a building).
  2. Generic Triple Filter: Using Natural Language Processing (NLP) and Part-of-Speech (POS) tagging, this filter identifies triples with low informational value. The heuristic is that if a subject or object is not a proper noun (e.g., "the company"), the statement is likely too generic for specific reasoning tasks.
  3. Factual Filter: Only triples that are both logically plausible and specific are sent for final factual verification against external sources.
The sequential multi-stage noise detection pipeline

Datasets and Evaluation

The approach was evaluated using a dataset of 2,342 manually labeled triples. Each was annotated for its noise type and truth value (True, False, or "True in the past"). The system leveraged external knowledge bases like DBpedia and YAGO to provide the necessary schema and axioms for the inconsistency detection stage.

Key Results

Preliminary experiments demonstrate that specialized filters are highly effective:

  • Inconsistent Detection: Achieved a Precision of 86.84% and a Recall of 62.26%. The high precision ensures that flagged triples are almost certainly noise, minimizing false positives.
  • Generic Detection: This stage was exceptionally successful. Using a combined NLP configuration, the system reached 98.25% Precision and 100% Recall, suggesting that identifying vague data is a largely solved problem with standard NLP tools.

Conclusion and Future Work

While the proposed pipeline offers a more granular approach to KG cleaning, challenges remain, particularly the lack of large-scale "gold standard" datasets for noise types. Future work will explore the use of graph embeddings to learn latent representations of entities and relations, potentially identifying inconsistencies that hardcoded ontological rules might miss. Additionally, the authors plan to use crowdsourcing to build more robust benchmarks for the research community.

Summary updated

BibTeX
@inproceedings{mihindukulasooriya2017towards,
  author = {Nandana Mihindukulasooriya and Oktie Hassanzadeh and Sarthak Dash and Alfio Gliozzo},
  title = {{Towards Comprehensive Noise Detection in Automatically Created Knowledge Graphs}},
  booktitle = {ISWC (Posters, Demos \& Industry Tracks)},
  year = {2017},
  url = {https://openreview.net/forum?id=PVLz_FIGiBg},
}
Figure