CapsTM: capsule network for Chinese medical text matching

Xiaoming Yu, Yedan Shen, Yuan Ni, Xiaowei Huang, Xiaolong Wang, Qingcai Chen, Buzhou Tang

Introduction

Text Matching (TM) is a critical component in medical Information Retrieval (IR) and Question Answering (QA) systems. The primary challenge lies in determining semantic equivalence between sentence pairs, such as "How do I treat Hepatitis B?" and "What is the therapy for Hep B?". While traditional deep learning models like Convolutional Neural Networks (CNNs) are effective, their reliance on pooling layers often results in the loss of spatial and structural information. This is particularly detrimental in the medical domain, where the precise relationship between symptoms, treatments, and conditions is paramount.

To address these limitations, researchers have introduced CapsTM, a novel architecture that integrates Capsule Networks into the text matching pipeline. By replacing conventional pooling with dynamic routing, CapsTM preserves the structural nuances of Chinese medical language.

Methodology

CapsTM is structured as a five-layer neural network designed to capture both local semantic features and global structural dependencies.

Architecture of CapsTM
  1. Input Layer: Text is converted into dense embeddings using word2vec or BERT. A "highway network" is employed to regulate the flow of information, mitigating the vanishing gradient problem.
  2. Representation Layer: Bi-directional Long Short-Term Memory (BiLSTM) networks capture sequential context. An attention-based interaction matrix is then computed to model the word-level relationships between the two input sentences.
  3. Aggregation Layer: A second BiLSTM layer fuses the individual sentence representations with the interactive features derived from the attention mechanism.
  4. Capsule Layer: This is the core innovation of CapsTM. Instead of scalar neurons and pooling, the model uses capsules—vectors that represent specific features. A dynamic routing algorithm iteratively determines the strength of connections between capsules in successive layers, effectively "voting" on the most relevant structural features.
  5. Prediction Layer: The final output is generated via a fully connected layer with a softmax function, providing a probability for the match/no-match classification.
Architecture of the Capsule Network and Dynamic Routing

Experimental Setup and Datasets

The model was evaluated on a specialized Chinese Medical Question Matching Corpus consisting of 36,360 manually annotated question pairs. The dataset was split into training (32,360), development (2,000), and test (2,000) sets. Experiments compared the performance of CapsTM using both standard word2vec and pre-trained BERT embeddings.

Key Results and Insights

CapsTM demonstrated superior performance over several strong baselines, including BIMPM and ESIM.

  • State-of-the-Art Performance: The BERT-integrated version of CapsTM achieved a top F-score of 0.8666, outperforming standalone BERT (0.8646).
  • The Value of Dynamic Routing: Ablation studies showed that replacing the Capsule Layer with standard max-pooling led to a performance decrease of 0.7% to 1.3%, validating the importance of the capsule architecture in preserving textual structure.
  • Interaction is Key: The attention mechanism proved to be the most critical component; its removal resulted in the most significant performance drop, highlighting the necessity of modeling sentence-to-sentence interactions.

Conclusion and Future Work

CapsTM successfully demonstrates that Capsule Networks can enhance the structural intelligence of text matching models in the medical domain. While the model excels at pattern recognition, it still faces challenges with fine-grained topic confusion and complex logical entailment. Future research will focus on integrating Clinical Knowledge Graphs to provide the model with explicit medical domain knowledge, moving beyond statistical matching toward deeper semantic understanding.

Summary updated

Abstract

Text Matching (TM) is a fundamental task of natural language processing widely used in many application systems such as information retrieval, automatic question answering, machine translation, dialogue system, reading comprehension, etc. In recent years, a large number of deep learning neural networks have been applied to TM, and have refreshed benchmarks of TM repeatedly. Among the deep learning neural networks, convolutional neural network (CNN) is one of the most popular networks, which suffers from difficulties in dealing with small samples and keeping relative structures of features. In this paper, we propose a novel deep learning architecture based on capsule network for TM, called CapsTM, where capsule network is a new type of neural network architecture proposed to address some of the short comings of CNN and shows great potential in many tasks.
BibTeX
@article{yu2021capstm,
  author = {Xiaoming Yu and Yedan Shen and Yuan Ni and Xiaowei Huang and Xiaolong Wang and Qingcai Chen and Buzhou Tang},
  title = {{CapsTM: capsule network for Chinese medical text matching}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2021},
  doi = {10.1186/s12911-021-01442-9},
}
Figure