TL;DR
This paper explores using Natural Language Processing (NLP) and machine learning (Random Forest and Convolutional Neural Networks) to automatically differentiate between three challenging inflammatory intestinal diseases (Ulcerative Colitis, Crohn's Disease, and Intestinal Tuberculosis) based on free-text endoscopic descriptions. The models achieved high accuracy, suggesting AI can significantly aid clinicians, especially those with less experience, in diagnosing these conditions.
What Problem Does This Paper Address?
Diagnosing inflammatory bowel diseases (IBD) like Ulcerative Colitis (UC) and Crohn's Disease (CD), alongside Intestinal Tuberculosis (ITB), is a significant clinical challenge. These conditions often present with similar symptoms and endoscopic appearances, leading to misdiagnoses and delayed treatment. This is particularly critical in regions like China, which has a high incidence of tuberculosis. While endoscopy is crucial for diagnosis, accurate differentiation relies heavily on the experience of the clinician. The paper aims to bridge this gap by developing automated diagnostic tools that can assist physicians, especially those less experienced, in distinguishing between these diseases.
Methods
The researchers employed a two-pronged machine learning approach: Random Forest (RF) and Convolutional Neural Networks (CNNs).
Data Preprocessing: 1. Word Segmentation: Free-text endoscopic descriptions were tokenized using a Chinese NLP package and a specialized medical dictionary, followed by the removal of punctuation and non-clinical terms. 2. Keyword Filtering: Term Frequency-Inverse Document Frequency (TF-IDF) was used to identify and filter for informative keywords within the descriptions. Terms with TF-IDF values outside a specific range ([0.3, 0.7]) were removed. 3. Dimension Reduction: Non-negative Matrix Factorization (NMF) was applied to reduce the dimensionality of the extracted features, aiming to improve interpretability.
Model Development: * Random Forest (RF): RF was chosen for its ability to weigh feature importance. It was used for both two-class (e.g., UC vs. CD) and three-class (UC, CD, ITB) classifications. To handle imbalanced datasets, cost-sensitive learning was employed. Feature extraction involved identifying the top 50 features based on RF's variable importance, which were then reviewed by experienced clinicians. * Convolutional Neural Networks (CNNs): CNNs were selected for their capability to automatically extract features. For training the CNN, a large corpus of unlabeled endoscopic descriptions was used to train a GloVe model for word embedding, which then initialized the CNN's embedding layer. The CNN architecture was based on the Text-CNN model. The paper notes that CNNs were not applied to all two-class problems due to extremely unbalanced sample sizes.
The models were trained on 70% of the data and tested on the remaining 30%.
Datasets
The study utilized a dataset of 6399 consecutive patients who underwent colonoscopies at Peking Union Medical College Hospital between January 2008 and November 2018. The dataset comprised:
- Ulcerative Colitis (UC): 5128 patients
- Crohn's Disease (CD): 875 patients
- Intestinal Tuberculosis (ITB): 396 patients
The input data for the models were the free-text descriptions of colonoscopic images. Clinically confirmed diagnoses from the hospital information system served as labels. The study also used approximately 110,000 unlabeled descriptions to train the GloVe word embedding model for the CNN.
Key Insights and Ideas
- NLP as a Diagnostic Aid: The study demonstrates that NLP can effectively extract meaningful features from unstructured clinical text (endoscopic descriptions) for disease classification, reducing reliance on manual feature engineering.
- CNNs Outperform RF in Specific Tasks: While both models performed well, CNNs generally showed better performance, particularly in differentiating CD and ITB.
- RF Uncovers Potentially New Features: The RF model identified features that were either new or rarely noted in previous literature for differentiating these diseases. For example, "ulcers covered with white exudate" was identified as a sign of CD rather than ITB, and "mucosal congestion and loss of vascular textures" were considered signs of CD when differentiating from ITB, contrary to some previous associations with UC.
- Interpretability vs. Performance Trade-off: RF offers better interpretability by highlighting feature importance, which can help clinicians understand the basis of the diagnosis. CNNs, while achieving higher performance in some cases, lack this direct interpretability, making their outputs more of a reference than a fully explainable diagnosis.
- Addressing Data Imbalance: The study highlights the importance of techniques like cost-sensitive learning for RF to handle imbalanced datasets, which are common in clinical data.
Key Results
- Two-Class Classifiers:
- UC vs. CD: RF achieved sensitivity/specificity of 0.89/0.84.
- UC vs. ITB: RF achieved sensitivity/specificity of 0.83/0.82.
- CD vs. ITB: RF achieved sensitivity/specificity of 0.72/0.77, while CNN achieved 0.90/0.77.
- Three-Class Classifier (UC, CD, ITB):
- RF: Precision/Recall for UC, CD, ITB were 0.97/0.97, 0.65/0.53, and 0.68/0.76, respectively.
- CNN: Precision/Recall for UC, CD, ITB were 0.99/0.97, 0.87/0.83, and 0.52/0.81, respectively.
- Overall Performance: The CNN generally showed better performance, especially in the challenging CD vs. ITB differentiation. The study concluded that both RF and CNN classifiers demonstrated excellent performance, with high specificity and sensitivity achieved for CD vs. ITB differentiation.
Limitations and Future Work
- Exclusion of Behçet's Disease: The study did not include Behçet's disease, which can have overlapping symptoms, as it's considered a systemic disease with variable initial presentations.
- Subjectivity in Descriptions: Despite efforts to standardize terminology, some subjectivity in endoscopists' descriptions might still exist.
- Data Imbalance: The dataset was highly imbalanced (UC >> CD > ITB), which is a common challenge in clinical research. While addressed, it remains a factor.
- Computational Resources: Classifiers directly analyzing endoscopic images (rather than text descriptions) would require significantly more computational power.
- Interpretability of CNNs: The "black box" nature of CNNs makes it difficult to understand why a particular diagnosis is made, limiting its immediate acceptance as a sole diagnostic tool.
- Future Directions: The authors suggest that direct image-based classification would be beneficial. They also propose that further clinical trials and larger cohort studies could improve the credibility of AI models. The potential for NLP models to be applied to other languages is also noted.
Summary updated