Complementary and Integrative Health Lexicon (CIHLex) and Entity Recognition in the Literature

Rui Zhang, PhD, Facmi, Famia, FiahsiarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces CIHLex, the first comprehensive lexicon for Complementary and Integrative Health (CIH) approaches, particularly focusing on physical and psychological aspects. They then leverage this lexicon to evaluate advanced Natural Language Processing (NLP) models, finding that BERT-based models significantly outperform traditional methods for recognizing CIH entities in biomedical literature.

What Problem Does This Paper Address?

Complementary and Integrative Health (CIH) approaches are increasingly used, but their documentation in electronic health records (EHRs) and biomedical literature is often inconsistent and poorly represented in standard terminologies. This makes it difficult to accurately track, research, and understand the prevalence and impact of CIH. Existing medical terminologies like the Unified Medical Language System (UMLS) have gaps in covering CIH concepts, especially for physical and psychological modalities. Furthermore, accurately extracting these CIH terms from unstructured text using NLP is a challenge due to the varied terminology and potential ambiguity.

Methods

The researchers developed the Complementary and Integrative Health Lexicon (CIHLex) by integrating terms from literature reviews and a specialized online resource (Natural Medicines). This involved compiling, de-duplicating, and manually validating terms to create 198 unique concepts and 1090 associated terms.

They then evaluated the coverage of CIHLex within the UMLS, mapping concepts and terms to identify existing representations and gaps.

For Named Entity Recognition (NER), they created a manually annotated corpus of PubMed abstracts related to CIH. This corpus was used to train and evaluate several NLP models, including:

  • Traditional NLP systems: MetaMap and CLAMP.
  • BERT-based models: Several fine-tuned versions of BERT, including BioBERT, ClinicalBERT, PubMedBERT, and BLUEBERT.
  • Large Language Models (LLMs): GPT-3.5 Turbo (tested in a zero-shot setting).

The models were evaluated using precision, recall, and F1-score, with both strict and lenient matching criteria.

Datasets

  • CIHLex: A newly constructed lexicon of 198 unique concepts and 1090 terms related to CIH, derived from literature reviews and the Natural Medicines database.
  • PubMed Abstracts: A corpus of 400 abstracts related to CIH, manually annotated for CIH entities. This corpus was split into training (80%), development (10%), and test (10%) sets, resulting in 2553 individual sentences for training BERT models.
  • UMLS Metathesaurus (2021AB version): Used for evaluating the coverage of CIHLex concepts and terms.

Key Insights and Ideas

  • CIH Terminology is Fragmented: The creation of CIHLex highlights the significant lack of standardized and comprehensive terminology for CIH approaches, particularly for physical modalities, which are less represented in existing resources like UMLS.
  • BERT's Superiority in CIH NER: Advanced transformer-based models like BERT demonstrate a clear advantage over older NLP techniques and even GPT-3.5 Turbo in accurately identifying and extracting CIH entities from complex biomedical text.
  • UMLS Gaps are Significant: A substantial portion of CIHLex concepts and terms could not be directly mapped to the UMLS, underscoring the need for its expansion to better represent CIH.
  • Semantic Type Diversity: While "Therapeutic or Preventive Procedure" is the dominant semantic type for mapped CIH concepts, the wide variety of other semantic types complicates efforts to specifically target CIH within UMLS.

Key Results

  • CIHLex Coverage: 62.1% of CIHLex concepts and 26.2% of its terms could be mapped to the UMLS. Notably, 133 (67.1%) concepts had terms that were not found in UMLS.
  • UMLS Mapping: 75.7% of mapped UMLS Concept Unique Identifiers (CUIs) fell under the "Therapeutic or Preventive Procedure" semantic type.
  • NER Performance: BLUEBERT achieved the highest macro-averaged F1-score of 0.90 for CIH NER, outperforming other BERT models, traditional NLP systems, and GPT-3.5 Turbo. BERT-based models generally showed superior performance across various CIH categories.
  • GPT-3.5 Turbo: In a zero-shot setting, GPT-3.5 Turbo showed promising performance (average F1 of 0.60) compared to existing NLP systems, but it also exhibited a tendency to misinterpret non-CIH terms or misclassify CIH subgroups.

Limitations and Future Work

  • CIHLex Limitations: The lexicon did not capture relationships (e.g., subsumption) between concepts. Abbreviations not fully defined in CIHLex were included in UMLS mapping, potentially affecting accuracy.
  • GPT-3.5 Turbo Evaluation: The evaluation of GPT-3.5 Turbo was limited to zero-shot testing with a small number of prompts, which may underestimate its true capabilities.
  • Future Work: The authors suggest future research should focus on developing new concepts within standardized languages to address representation gaps, exploring few-shot learning for LLMs like GPT-3.5 Turbo, and building a knowledge graph for CIH approaches.

Summary updated

BibTeX
@misc{fiahsi2023complementary,
  author = {Rui Zhang, PhD, FACMI, FAMIA, FIAHSI},
  title = {{Complementary and Integrative Health Lexicon (CIHLex) and Entity Recognition in the Literature}},
  year = {2023},
  eprint = {2305.17353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.17353},
}

Loading related content...

Figure