RNN Embeddings for Identifying Difficult to Understand Medical Words

Hanna Pylieva, Artem Chernodub, Natalia Grabar, Thierry HamonACL 2019

TL;DR

This paper tackles the challenge of making medical information accessible to non-experts by developing a novel RNN-based embedding method, FrnnMUTE, to identify difficult medical words. Their approach significantly improves the accuracy of classifying word understandability compared to existing methods, offering a valuable tool for enhancing health literacy.

What Problem Does This Paper Address?

Medical professionals use specialized terminology that can be a significant barrier for patients and their families trying to understand their health conditions, treatments, and procedures. While traditional readability metrics exist, they often rely on shallow linguistic features and are less effective for specialized texts like medical documents. This gap in understanding can lead to poorer health outcomes. The authors aim to bridge this gap by creating a system that can automatically identify which medical words are likely to be difficult for a layperson to comprehend.

Methods

The core of this paper is the development of a new type of word embedding called FrnnMUTE (French RNN Medical Understandability Text Embeddings). They trained a character-level Recurrent Neural Network (RNN) on French medical terms, using a sequence-to-sequence approach. The RNN was trained to classify words based on their understandability by a single annotator, and the final hidden state of the RNN was then used to generate a 50-dimensional vector representation for each word.

These FrnnMUTE embeddings were then used as features in a classification model, alongside standard linguistic features (like syntactic categories, word frequency in general and medical lexica, length, and morphological decomposition) and pre-trained FastText word embeddings. They explored different combinations of these features and evaluated their performance across three distinct cross-validation scenarios to assess generalization to new users and new vocabulary.

Datasets

The primary dataset used was a collection of biomedical terms obtained from the French part of Snomed International. After processing and filtering, they obtained 29,641 unique words. These words were then manually annotated by three French speakers without medical training. The annotators categorized each word into one of three levels: "I can understand the word," "I am not sure about the meaning," or "I cannot understand the word." The inter-annotator agreement was reported as substantial (Fleiss' Kappa 0.735, Cohen's Kappa 0.736).

Key Insights and Ideas

  • RNN Embeddings for Understandability: The core idea is that RNNs, by processing sequences of characters, can capture nuanced morphological and semantic information relevant to word difficulty, which simpler methods miss.
  • FrnnMUTE Outperforms FastText: The novel FrnnMUTE embeddings, trained specifically on medical term understandability, consistently performed better than generic FastText embeddings when used alone for classification. This highlights the benefit of domain-specific training.
  • Combination is Key (Mostly): Combining FrnnMUTE with standard linguistic features generally yielded the best results, suggesting that a multi-faceted approach is most effective. This indicates that while embeddings capture important information, traditional linguistic features still provide valuable context.
  • Generalization Matters: The different cross-validation scenarios revealed that models need to generalize well to both new users and new vocabulary. The FrnnMUTE embeddings showed more robust generalization across these scenarios compared to FastText.
  • Overfitting Concerns: The paper notes that combining all feature sets (standard, FastText, and FrnnMUTE) sometimes performed worse than using just standard and FrnnMUTE. This suggests a potential for overfitting when the feature dimensionality becomes too high, especially with the largest combination.

Key Results

The FrnnMUTE embeddings, when used in combination with standard features, achieved the highest F1 scores across various cross-validation scenarios. For instance, in the user-out vocabulary-out scenario, the combination of standard features and FrnnMUTE achieved a maximum F1 score of 87.0. When FrnnMUTE was used alone, it achieved a maximum F1 score of 82.4 in the user-out vocabulary-in scenario, significantly outperforming FastText embeddings alone (max F1 of 75.3 in the same scenario). The FrnnMUTE embeddings also demonstrated lower dispersion (standard deviation) across users, indicating better robustness.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Pre-trained Embeddings: The current FrnnMUTE embeddings are trained on a specific French medical dataset. Training word embeddings directly on medical data might further improve performance.
  • Nonlinear Dependencies: They hypothesize a robust nonlinear dependency between standard features and subword components of FastText embeddings, which they plan to investigate further.
  • Annotator Learning: The current experiments do not account for the learning progress annotators might show over time, which could influence the annotations. Future work could incorporate this dynamic aspect.
  • Data Availability: The FrnnMUTE embeddings and categorization code are being made publicly available for non-commercial research, which is a positive step for the community.

Summary updated

BibTeX
@inproceedings{pylieva2019rnn,
  author = {Hanna Pylieva and Artem Chernodub and Natalia Grabar and Thierry Hamon},
  title = {{RNN Embeddings for Identifying Difficult to Understand Medical Words}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure