Incorporating Medical Knowledge to Transformer-based Language Models for Medical Dialogue Generation

Usman Naseem, Ajay Bandi, Shaina Raza, Junaid Rashid, Bharathi Raja ChakravarthiACL 2022

TL;DR

This paper introduces a novel method to enhance transformer-based language models for medical dialogue generation by integrating external medical knowledge graphs. By injecting structured medical knowledge into the dialogue context, their approach significantly improves the accuracy and clinical relevance of generated responses, outperforming existing state-of-the-art models.

What Problem Does This Paper Address?

Medical dialogue systems hold immense potential for improving healthcare access, patient experience, and reducing costs by automating parts of the diagnostic process. However, current transformer-based language models, while powerful, often lack the specific domain knowledge required for accurate medical reasoning. This deficiency leads to dialogues that may not be medically sound or coherent with established medical knowledge, hindering their practical application in clinical settings. The challenge lies in equipping these models with the nuanced understanding that human doctors possess, enabling them to generate not just grammatically correct, but also clinically relevant and reliable responses.

Methods

The core of this paper's contribution is a novel architecture that incorporates a "knowledge layer" into transformer-based language models. This layer works by:

  1. Knowledge Graph Integration: They leverage an external medical knowledge graph (KG) containing entities (like organs, diseases) and their relationships (like function, treatment).
  2. K-Query and K-Inject: For an input utterance, the system first identifies relevant entities and queries the KG for associated triples (K-Query). These triples are then injected into the utterance, creating a "knowledge-rich utterance tree" (K-Inject).
  3. Masked Self-Attention: The utterance tree is then processed by an embedding layer and fed into a transformer encoder. Crucially, they employ a "masked self-attention" mechanism, an extension of standard self-attention, designed to prevent false semantic shifts and better preserve the original utterance's meaning while processing the enriched input.
  4. Transformer Decoder: Finally, a standard transformer decoder generates the response based on the enriched representation from the encoder and the dialogue history.

This approach aims to imbue the model with domain-specific knowledge without the need for expensive pre-training on massive, domain-specific corpora.

Datasets

The primary dataset used for training and evaluation is the MedDialog-EN dataset, which is the English version of a medical dialogue dataset. The paper notes that this dataset contains a substantial number of dialogues and utterances, making it suitable for training complex models. No new datasets were introduced in this work.

Key Insights and Ideas

  • Structured Knowledge is Key: The paper strongly suggests that simply relying on large language models trained on general text is insufficient for specialized domains like medicine. Explicitly incorporating structured knowledge (like from KGs) is crucial for achieving domain-specific competence.
  • Knowledge Injection as a Bridge: The proposed knowledge layer acts as an effective bridge, translating unstructured dialogue into a format that can be enriched by structured medical knowledge, thereby enhancing the model's understanding.
  • Masked Self-Attention for Preservation: The use of masked self-attention is a subtle but important detail. It highlights the need to balance the integration of new information with the preservation of the original semantic intent, a common challenge in sequence-to-sequence tasks.
  • Efficiency Gains: By injecting knowledge into existing pre-trained models rather than re-training from scratch, this method offers a more computationally efficient way to achieve domain expertise.

Key Results

The paper reports significant improvements across multiple evaluation metrics:

  • Automated Evaluation: Adding medical knowledge consistently improved performance across various metrics (BLEU2, BLEU4, METEOR, NIST-4) for all tested baseline models (BERT-GPT, Transformer, mT5, BART, T5). For instance, the performance increase for BART ranged from 1.33% to 1.35% for BLEU scores, and a substantial 1.03% increase for METEOR. For T5, the performance increase was even more pronounced, with an 8.15% jump in BLEU2 and 7.17% in BLEU4.
  • Human Evaluation: A panel of five medical doctors rated the generated responses. The results showed that responses generated by models incorporating medical knowledge were rated higher on a 1-to-5 scale, indicating they were more accurate, clinically informative, and human-like.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method improves performance, it is still an ongoing area of research. The complexity of medical dialogue and the vastness of medical knowledge mean that achieving human-level diagnostic capabilities is a long-term goal. The current KG used is focused on organs and related disorders, implying that broader medical knowledge might require more comprehensive KGs.
  • Future Work: The authors express their intention to expand this work to other tasks and datasets. This suggests potential applications beyond dialogue generation, such as medical question answering or summarization, and the exploration of different medical domains or knowledge sources.

Summary updated

BibTeX
@inproceedings{naseem2022incorporating,
  author = {Usman Naseem and Ajay Bandi and Shaina Raza and Junaid Rashid and Bharathi Raja Chakravarthi},
  title = {{Incorporating Medical Knowledge to Transformer-based Language Models for Medical Dialogue Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure