Tiny But Mighty: A Crowdsourced Benchmark Dataset for Triple Extraction from Unstructured Text

Muhammad Salman, Armin Haller, Sergio J. Rodriguez Mendez, Usman NaseemCOLING 2024

TL;DR

This paper introduces "Tiny But Mighty," a novel, crowdsourced benchmark dataset for evaluating knowledge graph (KG) construction from unstructured text. By addressing the lack of standardized evaluation datasets, this work provides a crucial resource for advancing KG construction techniques and Large Language Models (LLMs).

What Problem Does This Paper Address?

Constructing Knowledge Graphs (KGs) from unstructured text is a vital task for organizing information and extracting structured knowledge. However, a significant bottleneck in this field is the absence of standardized, high-quality benchmark datasets. This lack of a common ground makes it difficult to rigorously evaluate and compare different triple extraction methods. Existing approaches often rely on manually curated structured data or ad-hoc evaluations, which don't accurately reflect the challenges of real-world unstructured text. This paper aims to bridge this gap by introducing a meticulously annotated and verified dataset specifically designed for this purpose.

Methods

The core of this work lies in the creation of the "Tiny But Mighty" dataset. The process involved several key steps:

  • Leveraging Existing Resources: The authors started with the "Small But Mighty" dataset, a collection of compound and complex sentences from Wikipedia, originally designed for text simplification tasks.
  • Sentence Simplification: To make the annotation process more manageable, the complex sentences were first simplified. This was achieved using OpenAI's GPT-3.5, aiming to retain the original meaning while improving readability. This step is crucial for ensuring annotators can focus on extracting triples without being overwhelmed by syntactic complexity.
  • Crowdsourced Annotation (Phase 1): A large group of 127 volunteer annotators participated in a multi-round annotation process using a custom-built web-based tool. They were tasked with identifying subject-predicate-object triples from the simplified sentences. The process involved clear guidelines and iterative refinement over four rounds to improve consistency.
  • Expert Verification (Phase 2): To ensure high quality and accuracy, a subset of the annotated triples was rigorously verified by expert ontologists. This phase involved assessing the correctness, partial correctness, or nonsensical nature of the triples, leading to a high verified accuracy rate.
  • Novel Evaluation Criterion: The paper also proposes a triple-similarity evaluation criterion, the F-Measure, which is designed to handle the inherent differences between extracted triples from unstructured text and ground-truth triples. This metric incorporates penalties and amnesties based on similarity thresholds to provide a more nuanced evaluation.

Datasets

  • "Small But Mighty" Dataset: Used as the source for complex sentences.
  • "Tiny But Mighty" Dataset (Newly Introduced): This is the primary contribution. It consists of:
  • 720 original complex sentences.
  • 2,277 simplified sentences derived from the complex ones.
  • Over 11,000 annotated triples (final round: 3,736 triples for 720 sentences).
  • The dataset is enriched with expert-verified annotations, achieving a 96.22% success rate in Phase 2.

Key Insights and Ideas

  • The Importance of Simplification: Simplifying complex sentences before annotation significantly aids annotators, leading to more focused and accurate triple extraction. This highlights a practical approach to tackling the inherent complexity of natural language.
  • Crowdsourcing with Expert Validation is Key: Combining the scale of crowdsourcing with the rigor of expert ontologists is essential for creating high-quality, reliable datasets for NLP tasks. The multi-phase approach ensures both breadth and depth of annotation quality.
  • A Nuanced Evaluation is Necessary: Standard exact matching is insufficient for evaluating triple extraction from unstructured text. The proposed F-Measure with penalty/amnesty mechanisms offers a more robust way to assess the quality and relevance of extracted triples.
  • LLMs Can Benefit Greatly: The authors posit that this dataset can significantly enhance the fine-tuning of Large Language Models (LLMs) for KG construction, leading to more accurate and contextually aware triple generation.

Key Results

  • The "Tiny But Mighty" dataset comprises 720 complex and 2,277 simplified sentences with high-quality annotations.
  • Expert verification in Phase 2 achieved a success rate of 96.22% for the annotated triples.
  • Baseline models (SpaCy's SVO, CoreNLP OpenIE, and GPT-4) were evaluated on the dataset. GPT-4 generally led in performance across various similarity coefficients and evaluation criteria, demonstrating the dataset's utility in comparing models.
  • The F-Measure evaluation criterion, incorporating penalties and amnesties, showed that GPT-4 achieved the highest scores, indicating its superior ability to extract relevant and accurate triples.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset, while sufficient for distinguishing benchmark algorithms, is relatively small compared to other text corpora. This might affect the generalization capabilities when training KG construction models that require vast amounts of data. They plan to use this dataset to fine-tune LLMs to improve their ability to annotate larger texts.
  • RDF Transformation: The current dataset focuses on extracting triples (subject, predicate, object) but does not directly deal with RDF triples. The authors intend to transform predicates according to RDF standards in a future release.
  • Integration with External Knowledge: A future research direction is to investigate how KG construction from unstructured text can be improved by incorporating external knowledge sources, such as domain-specific ontologies or open-domain knowledge bases, potentially by extending the dataset with Wikidata mappings.

Summary updated

BibTeX
@inproceedings{salman2024tiny,
  author = {Muhammad Salman and Armin Haller and Sergio J. Rodriguez Mendez and Usman Naseem},
  title = {{Tiny But Mighty: A Crowdsourced Benchmark Dataset for Triple Extraction from Unstructured Text}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure