Towards Using Automatically Enhanced Knowledge Graphs to Aid Temporal Relation Extraction

Timotej Knez, Slavko ŽitnikCOLING 2024

TL;DR

This paper tackles the challenge of temporal relation extraction in medical documents by proposing a novel bimodal approach that combines textual information with knowledge graphs. By augmenting existing knowledge graphs with automatically generated common sense knowledge and patient-specific data, the authors achieve state-of-the-art results on a medical discharge summary dataset, demonstrating the power of enriched knowledge for improved temporal understanding.

What Problem Does This Paper Address?

Understanding the timeline of events in medical documents is crucial for patient care, diagnosis, and treatment. Temporal relation extraction (TRE) aims to identify these temporal associations between events described in text. However, existing TRE models often struggle with the nuances of medical language and the need for specialized domain knowledge. While knowledge graphs (KGs) offer a way to inject this external information, creating comprehensive and relevant KGs for medical TRE is a significant challenge. This paper addresses the limitations of existing KGs by proposing methods to automatically enhance them with common sense knowledge and patient-specific data, thereby improving TRE performance.

Methods

The core of the proposed approach is a bimodal model that integrates two sources of information:

  • Textual Content: An EntityBERT encoder processes the text of medical documents, generating embeddings for event mentions.
  • Knowledge Graph: A graph encoder, utilizing a convolutional graph neural network (GNN), aggregates information from a knowledge graph centered around event nodes.

These two types of embeddings are then merged to predict the temporal relation between events. The novelty lies in how the knowledge graph is constructed and enriched:

  1. PrimeKG Integration: The model leverages PrimeKG, a broad knowledge graph focused on precision medicine. To link text to PrimeKG, UMLS concepts are identified, and the relevant subgraph around matching nodes is extracted.
  2. Automatic Knowledge Graph Construction: Recognizing that PrimeKG lacks coverage for many medical concepts, the authors employ a large language model (OpenChat) to automatically generate knowledge graph triplets. This process involves prompting the LLM to create relations between medical concepts, significantly expanding the KG's coverage.
  3. Patient-Specific Knowledge Graph: To capture individual patient histories, a patient-specific graph is constructed by predicting relations between events within a single discharge summary. This provides temporal context specific to the patient.

The final knowledge graph used by the model is a combination of these three sources.

Datasets

The primary dataset used for evaluation is the i2b2 2012 dataset. This dataset consists of medical discharge summaries and contains 14,256 training relations and 11,752 testing relations for temporal relation extraction. The authors do not introduce a new benchmark dataset but rather focus on improving performance on an existing one.

Key Insights and Ideas

  • Common Sense is Key: Injecting common sense knowledge, even in a simplified form, significantly boosts temporal relation extraction performance, especially in specialized domains like medicine.
  • LLMs as KG Augmenters: Large language models are powerful tools for automatically expanding the coverage of knowledge graphs, addressing the limitations of manually curated KGs that may not encompass all relevant concepts.
  • Patient-Specific Context Matters: Incorporating patient-specific temporal information, derived from their own medical records, provides crucial context that improves the model's ability to understand event sequences.
  • Bimodal Approach Wins: Combining textual information with structured knowledge from graphs offers a more robust and accurate approach to temporal relation extraction than relying on text alone.

Key Results

The paper reports significant improvements in temporal relation extraction performance.

  • Adding a patient-specific knowledge graph to an EntityBERT encoder improved the F1 score by 1 percentage point, a statistically significant improvement (P-value of 0.039).
  • The proposed bimodal model, when enhanced with enriched knowledge graphs, surpassed other state-of-the-art models on the i2b2 2012 dataset.
  • Analysis of knowledge graph construction revealed that while PrimeKG links to only 22% of concepts from the i2b2 dataset, automatically generated graphs achieve 100% coverage of events.

Limitations and Future Work

  • Resource Intensive LLM Generation: The automatic generation of knowledge graphs using LLMs, while effective, requires significant computational resources, making it potentially impractical for some applications. The authors suggest generating these graphs in advance.
  • LLM-Generated Graph Quality: While LLM-generated graphs offer broader coverage, they may contain errors as they are not human-curated.
  • Ongoing Research: The authors emphasize that their research is ongoing. They plan to further evaluate the model's performance using the proposed automatically enhanced knowledge graph in future work. They also suggest that incorporating additional general knowledge beyond what was tested could lead to even greater improvements.

Summary updated

BibTeX
@inproceedings{knez2024towards,
  author = {Timotej Knez and Slavko Žitnik},
  title = {{Towards Using Automatically Enhanced Knowledge Graphs to Aid Temporal Relation Extraction}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure