Detection of medical text semantic similarity based on convolutional neural network

Tao Zheng, Yimei Gao, Fei Wang, Chenhao Fan, Xingzhi Fu, Mei Li, Ya Zhang, Shaodian Zhang, Handong Ma

Introduction

In clinical practice, "closing the loop" between diagnostic imaging and pathology is essential for quality assurance. Radiologists and pathologists often evaluate the same patient, yet their findings are frequently siloed within Electronic Medical Records (EMRs). To ensure diagnostic accuracy, hospital staff must manually compare imaging reports (e.g., an ultrasound of a neck mass) with subsequent pathology reports (e.g., a biopsy of the parotid gland).

This manual process is hindered by linguistic variability; one report might refer to the "maxillofacial region" while another specifies the "parotid gland." Traditional keyword matching fails to recognize that the parotid gland is a subset of the maxillofacial region. This study addresses the challenge of semantic similarity—identifying when disparate text descriptions refer to the same anatomical site.

Methods

The researchers developed an end-to-end Convolutional Neural Network (CNN) architecture designed to capture local, hierarchical patterns in medical text. Unlike generic NLP models, this approach integrates domain-specific knowledge through Ontology-Based Graph Embedding. By utilizing CMeSH (Chinese Medical Subject Headings) and graph embedding techniques, the model captures anatomical hierarchies before training begins.

CNN architecture for text similarity detection

The model employs a dual-input CNN structure that processes imaging and pathology reports through parallel convolutional layers. These layers extract feature vectors which are then concatenated to predict the probability of a match. To address the "black box" nature of neural networks, the study utilized the LIME (Local Interpretable Model-agnostic Explanations) algorithm to identify specific keywords, such as "thyroid" or "fetus," that most influenced the model's decisions.

Dataset and Evaluation

The study utilized a real-world dataset from Shanghai Tongren Hospital, consisting of 16,354 imaging and pathology report pairs from 1,926 patients collected over a three-month period in 2017. Ground truth was established by three physicians who manually annotated the pairs for anatomical overlap, achieving high inter-rater reliability (Kappa 0.95–0.97). The data was split into 80% for training and 20% for testing.

Key Results

The CNN model initialized with medical concept vectors significantly outperformed traditional NLP methods and Siamese LSTMs. The hierarchical nature of CNN kernels proved particularly effective at mapping specific tissues to broader anatomical regions.

ROC Curve comparing various models

As shown in the ROC analysis, the CNN model with concept vectors achieved an AUROC of 0.951, representing a 3–7% improvement over Latent Dirichlet Allocation (LDA) and Latent Semantic Analysis (LSA). The model also achieved a Macro F1-score of 0.935, a substantial increase over the keyword mapping baseline of 0.833. Even the simplest CNN version with random initialization outperformed complex traditional models, validating the architecture's suitability for medical text matching.

Discussion and Limitations

While the model demonstrates high accuracy, several challenges remain. The reliance on the Jieba segmentation tool for Chinese text can occasionally lead to errors in specialized medical terminology. Furthermore, the current model acts as a binary classifier; future iterations could implement a ranking system to help clinicians prioritize the most relevant reports. Finally, the model occasionally struggles with complex 3D spatial relationships (e.g., "ventral side of the inferior pole"), suggesting that incorporating more sophisticated spatial reasoning remains a key area for future research.

Summary updated

Abstract

Background Imaging examinations, such as ultrasonography, magnetic resonance imaging and computed tomography scans, play key roles in healthcare settings. To assess and improve the quality of imaging diagnosis, we need to manually find and compare the pre-existing reports of imaging and pathology examinations which contain overlapping exam body sites from electrical medical records (EMRs). The process of retrieving those reports is time-consuming. In this paper, we propose a convolutional neural network (CNN) based method which can better utilize semantic information contained in report texts to accelerate the retrieving process. Methods We included 16,354 imaging and pathology report-pairs from 1926 patients who admitted to Shanghai Tongren Hospital and had ultrasonic examinations between 1st May 2017 and 31st July 2017. We adapted the CNN model to calculate the similarities among the report-pairs to identify target report-pairs with overlapping body sites, and compared the performance with other six conventional models, including keyword mapping, latent semantic analysis (LSA), latent Dirichlet allocation (LDA), Doc2Vec, Siamese long short term memory (LSTM) and a model based on named entity recognition (NER). We also utilized graph embedding method to enhance the word representation by capturing the semantic relations information from medical ontologies. Additionally, we used LIME algorithm to identify which features (or words) are decisive for the prediction results and improved the model interpretability. Results Experiment results showed that our CNN model gained significant improvement compared to all other conventional models on area under the receiver operating characteristic (AUROC), precision, recall and F1-score in our test dataset. The AUROC of our CNN models gained approximately 3–7% improvement. The AUROC of CNN model with graph-embedding and ontology based medical concept vectors was 0.8% higher than the model with randomly initialized vectors and 1.5% higher than the one with pre-trained word vectors. Conclusion Our study demonstrates that CNN model with pre-trained medical concept vectors could accurately identify target report-pairs with overlapping body sites and potentially accelerate the retrieving process for imaging diagnosis quality measurement.
BibTeX
@article{zheng2019detection,
  author = {Tao Zheng and Yimei Gao and Fei Wang and Chenhao Fan and Xingzhi Fu and Mei Li and Ya Zhang and Shaodian Zhang and Handong Ma},
  title = {{Detection of medical text semantic similarity based on convolutional neural network}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2019},
  doi = {10.1186/s12911-019-0880-2},
}
Figure