Can natural language processing help differentiate inflammatory intestinal diseases in China? Models applying random forest and convolutional neural network approaches

Yuanren Tong, Keming Lu, Yingyun Yang, Ji Li, Yucong Lin, Dong Wu, Aiming Yang, Yue Li, Sheng Yu, Jiaming QianBMC Medical Informatics Decis. Mak 2020 · OpenReview

TL;DR

This paper explores using Natural Language Processing (NLP) and machine learning (Random Forest and Convolutional Neural Networks) to automatically differentiate between three challenging inflammatory intestinal diseases (Ulcerative Colitis, Crohn's Disease, and Intestinal Tuberculosis) based on free-text endoscopic descriptions. The models achieved high accuracy, suggesting AI can significantly aid clinicians, especially those with less experience, in diagnosing these conditions.

What Problem Does This Paper Address?

Diagnosing inflammatory bowel diseases (IBD) like Ulcerative Colitis (UC) and Crohn's Disease (CD), alongside Intestinal Tuberculosis (ITB), is a significant clinical challenge. These conditions often present with similar symptoms and endoscopic appearances, leading to misdiagnoses and delayed treatment. This is particularly critical in regions like China, which has a high incidence of tuberculosis. While endoscopy is crucial for diagnosis, accurate differentiation relies heavily on the experience of the clinician. The paper aims to bridge this gap by developing automated diagnostic tools that can assist physicians, especially those less experienced, in distinguishing between these diseases.

Methods

The researchers employed a two-pronged machine learning approach: Random Forest (RF) and Convolutional Neural Networks (CNNs).

Data Preprocessing: 1. Word Segmentation: Free-text endoscopic descriptions were tokenized using a Chinese NLP package and a specialized medical dictionary, followed by the removal of punctuation and non-clinical terms. 2. Keyword Filtering: Term Frequency-Inverse Document Frequency (TF-IDF) was used to identify and filter for informative keywords within the descriptions. Terms with TF-IDF values outside a specific range ([0.3, 0.7]) were removed. 3. Dimension Reduction: Non-negative Matrix Factorization (NMF) was applied to reduce the dimensionality of the extracted features, aiming to improve interpretability.

Model Development: * Random Forest (RF): RF was chosen for its ability to weigh feature importance. It was used for both two-class (e.g., UC vs. CD) and three-class (UC, CD, ITB) classifications. To handle imbalanced datasets, cost-sensitive learning was employed. Feature extraction involved identifying the top 50 features based on RF's variable importance, which were then reviewed by experienced clinicians. * Convolutional Neural Networks (CNNs): CNNs were selected for their capability to automatically extract features. For training the CNN, a large corpus of unlabeled endoscopic descriptions was used to train a GloVe model for word embedding, which then initialized the CNN's embedding layer. The CNN architecture was based on the Text-CNN model. The paper notes that CNNs were not applied to all two-class problems due to extremely unbalanced sample sizes.

The models were trained on 70% of the data and tested on the remaining 30%.

Datasets

The study utilized a dataset of 6399 consecutive patients who underwent colonoscopies at Peking Union Medical College Hospital between January 2008 and November 2018. The dataset comprised:

  • Ulcerative Colitis (UC): 5128 patients
  • Crohn's Disease (CD): 875 patients
  • Intestinal Tuberculosis (ITB): 396 patients

The input data for the models were the free-text descriptions of colonoscopic images. Clinically confirmed diagnoses from the hospital information system served as labels. The study also used approximately 110,000 unlabeled descriptions to train the GloVe word embedding model for the CNN.

Key Insights and Ideas

  • NLP as a Diagnostic Aid: The study demonstrates that NLP can effectively extract meaningful features from unstructured clinical text (endoscopic descriptions) for disease classification, reducing reliance on manual feature engineering.
  • CNNs Outperform RF in Specific Tasks: While both models performed well, CNNs generally showed better performance, particularly in differentiating CD and ITB.
  • RF Uncovers Potentially New Features: The RF model identified features that were either new or rarely noted in previous literature for differentiating these diseases. For example, "ulcers covered with white exudate" was identified as a sign of CD rather than ITB, and "mucosal congestion and loss of vascular textures" were considered signs of CD when differentiating from ITB, contrary to some previous associations with UC.
  • Interpretability vs. Performance Trade-off: RF offers better interpretability by highlighting feature importance, which can help clinicians understand the basis of the diagnosis. CNNs, while achieving higher performance in some cases, lack this direct interpretability, making their outputs more of a reference than a fully explainable diagnosis.
  • Addressing Data Imbalance: The study highlights the importance of techniques like cost-sensitive learning for RF to handle imbalanced datasets, which are common in clinical data.

Key Results

  • Two-Class Classifiers:
    • UC vs. CD: RF achieved sensitivity/specificity of 0.89/0.84.
  • UC vs. ITB: RF achieved sensitivity/specificity of 0.83/0.82.
  • CD vs. ITB: RF achieved sensitivity/specificity of 0.72/0.77, while CNN achieved 0.90/0.77.
  • Three-Class Classifier (UC, CD, ITB):
    • RF: Precision/Recall for UC, CD, ITB were 0.97/0.97, 0.65/0.53, and 0.68/0.76, respectively.
  • CNN: Precision/Recall for UC, CD, ITB were 0.99/0.97, 0.87/0.83, and 0.52/0.81, respectively.
  • Overall Performance: The CNN generally showed better performance, especially in the challenging CD vs. ITB differentiation. The study concluded that both RF and CNN classifiers demonstrated excellent performance, with high specificity and sensitivity achieved for CD vs. ITB differentiation.

Limitations and Future Work

  • Exclusion of Behçet's Disease: The study did not include Behçet's disease, which can have overlapping symptoms, as it's considered a systemic disease with variable initial presentations.
  • Subjectivity in Descriptions: Despite efforts to standardize terminology, some subjectivity in endoscopists' descriptions might still exist.
  • Data Imbalance: The dataset was highly imbalanced (UC >> CD > ITB), which is a common challenge in clinical research. While addressed, it remains a factor.
  • Computational Resources: Classifiers directly analyzing endoscopic images (rather than text descriptions) would require significantly more computational power.
  • Interpretability of CNNs: The "black box" nature of CNNs makes it difficult to understand why a particular diagnosis is made, limiting its immediate acceptance as a sole diagnostic tool.
  • Future Directions: The authors suggest that direct image-based classification would be beneficial. They also propose that further clinical trials and larger cohort studies could improve the credibility of AI models. The potential for NLP models to be applied to other languages is also noted.

Summary updated

Abstract

Background Differentiating between ulcerative colitis (UC), Crohn’s disease (CD) and intestinal tuberculosis (ITB) using endoscopy is challenging. We aimed to realize automatic differential diagnosis among these diseases through machine learning algorithms. Methods A total of 6399 consecutive patients (5128 UC, 875 CD and 396 ITB) who had undergone colonoscopy examinations in the Peking Union Medical College Hospital from January 2008 to November 2018 were enrolled. The input was the description of the endoscopic image in the form of free text. Word segmentation and key word filtering were conducted as data preprocessing. Random forest (RF) and convolutional neural network (CNN) approaches were applied to different disease entities. Three two-class classifiers (UC and CD, UC and ITB, and CD and ITB) and a three-class classifier (UC, CD and ITB) were built. Results The classifiers built in this research performed well, and the CNN had better performance in general. The RF sensitivities/specificities of UC-CD, UC-ITB, and CD-ITB were 0.89/0.84, 0.83/0.82, and 0.72/0.77, respectively, while the values for the CNN of CD-ITB were 0.90/0.77. The precisions/recalls of UC-CD-ITB when employing RF were 0.97/0.97, 0.65/0.53, and 0.68/0.76, respectively, and when employing the CNN were 0.99/0.97, 0.87/0.83, and 0.52/0.81, respectively. Conclusions Classifiers built by RF and CNN approaches had excellent performance when classifying UC with CD or ITB. For the differentiation of CD and ITB, high specificity and sensitivity were achieved as well. Artificial intelligence through machine learning is very promising in helping unexperienced endoscopists differentiate inflammatory intestinal diseases. Conference The abstract of this article has won the first prize of the Young Investigator Award during the Asian Pacific Digestive Week (APDW) 2019 held in Kolkata, India.
BibTeX
@inproceedings{tong2020can,
  author = {Yuanren Tong and Keming Lu and Yingyun Yang and Ji Li and Yucong Lin and Dong Wu and Aiming Yang and Yue Li and Sheng Yu and Jiaming Qian},
  title = {{Can natural language processing help differentiate inflammatory intestinal diseases in China? Models applying random forest and convolutional neural network approaches}},
  booktitle = {BMC Medical Informatics Decis. Mak},
  year = {2020},
  url = {https://openreview.net/forum?id=2Mewve0YwN},
}
Figure