KGAST: From Knowledge Graphs to Annotated Synthetic Texts

Nakanyseth Vuth, Gilles Sérasset, Didier SchwabACL 2024

TL;DR

This paper introduces KGAST, a novel framework that uses Knowledge Graphs (KGs) and Large Language Models (LLMs) to automatically generate synthetic text data with annotations for Information Extraction (IE) tasks. This approach aims to alleviate data scarcity and privacy concerns, showing promise in complementing existing datasets to improve model performance, especially for under-represented classes.

What Problem Does This Paper Address?

Information Extraction (IE) models are crucial for making sense of complex data across various domains. However, their effectiveness hinges on the availability of high-quality training data. Two major challenges hinder this: Data Scarcity, where datasets are often too small or entirely unavailable, preventing models from generalizing effectively, and Privacy and Compliance Concerns, where sensitive data cannot be shared due to regulations, further limiting training possibilities. While manual data augmentation is an option, it's labor-intensive and costly. Existing automatic methods like simple word replacement or back-translation often lack semantic diversity or introduce annotation misalignment, making them less effective for robust model training.

Methods

The KGAST framework leverages two core components: Large Language Models (LLMs) and In-Context Learning (ICL). Instead of modifying existing texts, KGAST generates entirely new synthetic texts and their corresponding annotations. It utilizes a Chain-of-Thoughts (CoT) prompting method to create complex ICL prompt templates for LLMs. The core idea is inspired by distant supervision: if two entities share a relation in a Knowledge Graph (KG), the framework assumes any generated text involving these entities should reflect that relation.

The process involves:

  1. KG as Starting Point: A KG ($K_t$) is used to retrieve relevant k-shot examples.
  2. Synthetic Text Generation: LLMs are prompted using these examples and the KG to generate synthetic texts ($t'$).
  3. Text Filtering/Voting: A voting mechanism is employed to select the "best" synthetic text ($t'$) based on criteria like creativity and coherence.
  4. Synthetic Annotation: The selected text ($t'$) is then annotated using another LLM prompt, drawing inspiration from Tree-of-Thought prompting.
  5. Annotation Merging: The LLM-generated annotations are merged with a "naive" heuristic (based on entity presence in the text) to produce the final synthetic dataset.
  6. Self-Consistency: To combat issues like incomplete annotations and hallucinations, a self-consistency approach is integrated, where multiple outputs are generated and the most consistent one is selected.

Datasets

The paper uses two datasets for experiments:

  • DocRED: A large-scale document-level relation extraction dataset constructed from Wikipedia and Wikidata, containing 96 relations and 6 entity types for English.
  • French Security and Defense (FRSD): A document-level relation extraction dataset for French, covering Event Extraction, Entity Recognition, Attribute Extraction, and Relation Extraction tasks. It contains 2,000 French documents, with 800 manually written and annotated.

The authors generated synthetic versions of these datasets using their KGAST framework.

Key Insights and Ideas

  • KG-driven Generation is Key: Using KGs as a structured source of information for LLMs is crucial for generating semantically relevant synthetic text and annotations.
  • Synthetic Data as a Complement: The study strongly suggests that synthetic data is most effective when used to complement existing gold-standard data, particularly for improving performance on classes with limited training samples.
  • LLM Capabilities are Paramount: The quality of the generated synthetic data and annotations is heavily dependent on the performance of the LLM used.
  • Annotation Noise is a Challenge: Even with sophisticated methods, synthetic annotations can still contain noise due to incomplete extraction or wrongly assumed relations, requiring careful validation.
  • Diversity Matters: The paper hints that higher semantic differences between gold and synthetic texts might lead to better generalization and more robust performance, suggesting avenues for future research in controlling this diversity.

Key Results

  • Intrinsic Evaluation: The KGAST framework achieved high F1-scores for Named Entity Recognition (NER) (0.93 and 0.92) and reasonable scores for Relation Extraction (RE) (0.63 F1-score for Relation, 0.37 for Evidence) on DocRED's synthetic data.
  • Extrinsic Evaluation (DocRED):
    • For NER, training solely on synthetic data (S) yielded acceptable results (79.14 Weighted-F1), but a significant performance gap existed compared to training on gold data (G). Combining G+S generally produced the best results.
  • For RE, training on synthetic data alone resulted in a notable drop in performance compared to gold data.
  • Extrinsic Evaluation (FRSD): Synthetic data (S) showed promise in enhancing performance for classes with fewer samples in Event and Attribute extraction tasks when used as a complement.
  • Semantic vs. Lexical Similarity: Analysis showed that while synthetic texts maintained semantic similarity with gold texts, lexical differences were more pronounced, indicating the generation of varied phrasing.

Limitations and Future Work

  • Data Diversity: The current framework's diversity is limited by the original KGs. Future work could involve experimenting with modified KGs to enhance diversity.
  • Annotation Imperfection: The generated annotations are not perfect and can contain noise.
  • Future Work Suggestions:
    • Using dependency trees to identify co-references and extract relations.
  • Leveraging attention weights from generated texts to pinpoint evidence of relations.
  • Exploring parameter control, prompt template refinement, or text filtering to increase diversity and reduce Self-BLEU scores.
  • Investigating methods to mitigate noise in synthetic annotations.

Summary updated

BibTeX
@inproceedings{vuth2024kgast,
  author = {Nakanyseth Vuth and Gilles Sérasset and Didier Schwab},
  title = {{KGAST: From Knowledge Graphs to Annotated Synthetic Texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure