Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations

Yiming Li, Xueqing Peng, Jianfu Li, Xu Zuo, Suyuan Peng, Donghong Pei, Cui Tao, Hua Xu, Na Hong

Abstract

In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPT) present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to compare the performance of GPT with traditional deep learning models (Long Short-Term Memory (LSTM) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT)) in extracting acupoint-related location relations and assess the impact of pretraining and fine-tuning on GPT's performance. We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations ('direction_of,' 'distance_of,' 'part_of,' 'near_acupoint,' and 'located_near') (n= 3,174) between acupoints were annotated. Five models were compared: BioBERT, LSTM, pre-trained GPT-3.5, fine-tuned GPT-3.5, as well as pre-trained GPT-4. Performance metrics included micro-average exact match precision, recall, and F1 scores. Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. This study underscores the effectiveness of LLMs like GPT in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

Introduction

Acupuncture's therapeutic efficacy is fundamentally tied to the precise identification of acupuncture points (acupoints). Traditionally, this knowledge is codified in descriptive texts that utilize a highly specialized spatial and anatomical vocabulary. For instance, a point might be described as being "3 B-cun inferior to SP9" or located on a specific "tibial aspect." For medical informatics, the challenge lies in Relation Extraction (RE): the process of converting these unstructured, complex textual descriptions into structured data that can be utilized by computational systems.

Traditional Natural Language Processing (NLP) models, such as LSTMs and even domain-specific models like BioBERT, often struggle with the intricate spatial dependencies and niche terminology found in traditional medicine. This study investigates whether the generative power and reasoning capabilities of Large Language Models (LLMs) can overcome these hurdles.

Methodology

The research team developed a comprehensive pipeline to compare traditional deep learning architectures against modern LLMs. The workflow involved extracting descriptions from standard texts, manual annotation, model training/fine-tuning, and rigorous evaluation.

The study workflow, illustrating the path from WHO standard texts to model fine-tuning and evaluation.

Five model configurations were evaluated:

  1. LSTM: A standard recurrent neural network baseline.
  2. BioBERT: A BERT variant pre-trained on large-scale biomedical corpora (e.g., PubMed).
  3. Pre-trained GPT-3.5-turbo: Evaluated in a zero-shot capacity using prompt tuning.
  4. Fine-tuned GPT-3.5-turbo: Specifically trained on a subset of annotated acupuncture data.
  5. Pre-trained GPT-4: Tested to determine if increased model scale could compensate for a lack of domain-specific fine-tuning.

Data and Annotation

The study utilized the World Health Organization (WHO) Standard Acupuncture Point Locations in the Western Pacific Region as the primary corpus. This dataset covers 361 acupoints. Researchers manually annotated 3,174 relation pairs using the CLAMP toolkit, focusing on five distinct spatial and anatomical relations.

Definitions and examples of the five relation types used to map acupoint locations.

The relations defined were:

  • direction_of: The direction of one entity relative to another.
  • distance_of: The specific measurement (e.g., B-cun) between entities.
  • part_of: Sub-anatomical relationships.
  • near_acupoint: Adjacency between two specific acupoints.
  • located_near: Proximity of an anatomy to an acupoint of interest.

Key Results

The fine-tuned GPT-3.5 model emerged as the superior performer across all metrics, achieving a micro-average F1 score of 0.92. In contrast, the pre-trained GPT-3.5 and GPT-4 models achieved F1 scores of only 0.41 and 0.32, respectively.

Notably, traditional models like BioBERT and LSTM were largely ineffective for this specific task, with F1 scores failing to exceed 0.20. This suggests that while BioBERT is powerful for general clinical text, it lacks the specific anatomical and spatial context required for acupuncture standards. The fine-tuned GPT-3.5 was particularly adept at identifying direction_of relations, reaching an F1 of 0.97, though all models found the near_acupoint relation more challenging due to its complex spatial context.

Discussion and Conclusion

The findings highlight a critical insight for medical NLP: for niche domains like traditional medicine, fine-tuning is non-negotiable. Even the significantly larger GPT-4 model, when used without fine-tuning, could not match the performance of a smaller, domain-adapted GPT-3.5.

Despite the success, challenges remain in "contextual blindness," where models struggle to track entities across multiple sentences or when descriptions are interrupted by secondary notes. Future research will focus on integrating structured anatomical knowledge bases to provide models with a better "visual" understanding of the human body. Ultimately, this work paves the way for more accurate digital modeling of acupuncture knowledge, supporting standardized training and clinical practice worldwide.

Summary updated

Abstract

In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPT) present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to compare the performance of GPT with traditional deep learning models (Long Short-Term Memory (LSTM) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT)) in extracting acupoint-related location relations and assess the impact of pretraining and fine-tuning on GPT's performance. We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations ('direction_of,' 'distance_of,' 'part_of,' 'near_acupoint,' and 'located_near') (n= 3,174) between acupoints were annotated. Five models were compared: BioBERT, LSTM, pre-trained GPT-3.5, fine-tuned GPT-3.5, as well as pre-trained GPT-4. Performance metrics included micro-average exact match precision, recall, and F1 scores. Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. This study underscores the effectiveness of LLMs like GPT in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.
BibTeX
@misc{li2024relation,
  author = {Yiming Li and Xueqing Peng and Jianfu Li and Xu Zuo and Suyuan Peng and Donghong Pei and Cui Tao and Hua Xu and Na Hong},
  title = {{Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations}},
  year = {2024},
  eprint = {2404.05415},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05415},
}
Figure