Improving Medical Short Text Classification with Semantic Expansion Using Word-Cluster Embedding

Ying Shen, Qiang Zhang, Jin Zhang, Jiyue Huang, Yuming Lu, Kai Lei

Abstract

Classifying medical short text, such as electronic medical records (EMRs), is challenging due to feature sparseness and semantic ambiguity. This paper introduces a "Word-Cluster Embedding" framework that expands standard word vectors with implicit topic information derived from Hierarchical Agglomerative Clustering (HAC). By concatenating word embeddings with their corresponding cluster center vectors, the model provides richer context for deep learning classifiers. Our results show that the HAC-LSTM model achieves an accuracy of 0.947 on the Chinese Medical Short Sentence (CMSS) dataset, outperforming existing state-of-the-art methods.

Introduction

The digitization of healthcare has led to a surge in Electronic Medical Records (EMRs). However, these records often consist of short, fragmented sentences that present significant hurdles for automated classification. Two primary issues are feature sparseness—where the limited number of words provides insufficient context—and semantic ambiguity, where synonyms, acronyms, and informal medical slang describe the same underlying condition.

While deep learning models like CNNs and LSTMs have improved text classification, they often struggle with short inputs that lack a clear "topic." This research proposes bridging this gap by injecting "implicit topic information" into every word vector through a process of semantic expansion.

Methodology: Word-Cluster Embedding

The proposed framework enhances traditional word embeddings by incorporating broader semantic context through clustering. The process involves three critical stages:

  1. Word Vector Generation: Dense, low-dimensional vectors are generated using a Skip-Gram model (Word2Vec) trained on a massive medical corpus.
  2. Hierarchical Agglomerative Clustering (HAC): To capture "implicit topics," word vectors are grouped using HAC with average linkage. This method was selected for its ability to produce cohesive, granular clusters that are less sensitive to outliers compared to algorithms like DBSCAN or APC.
  3. Semantic Expansion: Each word vector is concatenated with the center vector of its assigned cluster. This combined "Word-Cluster Embedding" serves as the input for the classification layer.
The Word-Cluster Embedding generation process combines Skip-Gram word vectors with cluster center vectors derived from labeled corpora.

The final classification is performed using standard deep learning architectures, specifically Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks, which leverage the expanded semantic information to categorize medical symptoms.

Experimental Setup

The model was evaluated using two primary datasets:

  • CMSS (Chinese Medical Short Sentence): A custom dataset of 17,787 sentences categorized by symptom severity (slightly, moderately, heavily).
  • TREC: A standard question-classification dataset (6,000 samples) used to verify the model's generalizability outside the medical domain.

To ensure robust embeddings, the authors pre-trained the model on a corpus of over 430 million words, including medical entries from Baidu Encyclopedia, Chinese Wikipedia, and anonymized EMRs.

Key Results and Discussion

The core innovation of this work is treating cluster centers as proxies for topics. For instance, a word like "flu" is enriched by a cluster center containing related terms like "fever" or "cough," effectively signaling the broader clinical context to the model even if those terms are absent from the specific short text.

The HAC-LSTM model demonstrated superior performance across both medical and general datasets. On the CMSS dataset, it achieved a state-of-the-art accuracy of 0.947.

Comparison of accuracy across different models on the CMSS dataset, showing the superiority of the HAC-LSTM approach.

Comparative analysis revealed that HAC outperformed other clustering methods like Affinity Propagation (APC) and DBSCAN. The bottom-up nature of HAC is particularly suited for creating the small, semantic-heavy clusters required for word-level expansion.

Conclusion and Future Work

The Word-Cluster Embedding approach effectively mitigates the limitations of short text classification in the medical domain. While the model handles medical slang and synonyms well, challenges remain regarding word segmentation in Chinese medical text. Future research will explore the integration of Named Entity Recognition (NER) to better identify complex medical entities and further refine the clustering process.

Summary updated

Abstract

Automatic text classification (TC) research can be used for real-world problems such as the classification of in-patient discharge summaries and medical text reports, which is beneficial to make medical documents more understandable to doctors. However, in electronic medical records (EMR), the texts containing sentences are shorter than that in general domain, which leads to the lack of semantic features and the ambiguity of semantic. To tackle this challenge, we propose to add word-cluster embedding to deep neural network for improving short text classification. Concretely, we first use hierarchical agglomerative clustering to cluster the word vectors in the semantic space. Then we calculate the cluster center vector which represents the implicit topic information of words in the cluster. Finally, we expand word vector with cluster center vector, and implement classifiers using CNN and LSTM respectively. To evaluate the performance of our proposed method, we conduct experiments on public data sets TREC and the medical short sentences data sets which is constructed and released by us. The experimental results demonstrate that our proposed method outperforms state-of-the-art baselines in short sentence classification on both medical domain and general domain.
BibTeX
@inproceedings{shen2018improving,
  author = {Ying Shen and Qiang Zhang and Jin Zhang and Jiyue Huang and Yuming Lu and Kai Lei},
  title = {{Improving Medical Short Text Classification with Semantic Expansion Using Word-Cluster Embedding}},
  booktitle = {ICISA},
  year = {2018},
  url = {https://openreview.net/forum?id=vBOnuW7IG0},
}
Figure