Introduction
Traditional Chinese Medicine (TCM) relies heavily on the concept of "Constitution" (TCMC), which characterizes an individual's physiological state and susceptibility to diseases. Traditionally, TCMC is assessed through four diagnostic methods: inspection, listening/smelling, inquiry, and pulse-taking. Automating this process presents significant challenges, primarily in capturing nuanced clinical features (like tongue luster or facial color) and effectively fusing disparate data sources such as visual imagery and structured questionnaire data.
The TCM-CAS Architecture
To address these challenges, researchers presented TCM-CAS at AAAI-22. This end-to-end system automates TCMC assessment by integrating multimodal deep learning techniques. The architecture is divided into four primary modules: Face, Tongue-Dorsal, Question, and Prediction.
The Face Module utilizes the Mediapipe framework for landmark detection and alignment, followed by segmentation to extract TCM-relevant features like lip and facial color. Similarly, the Tongue-Dorsal Module employs the MiniSeg model for precise tongue segmentation, focusing on tongue body and coating characteristics.
Graph-Based Symptom Modeling
The most innovative component of TCM-CAS is the Question Module. Rather than treating questionnaire responses as isolated text strings, the system constructs a heterogeneous graph where nodes represent patients and symptoms. Edges in the graph denote the co-occurrence probability of symptoms, allowing the model to capture complex, holistic relationships between physical indicators.
By applying GraphSAGE, the system learns rich embeddings for these nodes. This approach mimics the reasoning of human TCM practitioners who look for patterns of symptoms rather than individual data points.
Results and Performance
The system was evaluated on a dataset of 5,515 samples across four TCMC categories. The results demonstrate the superiority of the multimodal approach:
- State-of-the-Art Accuracy: The
GraphSAGE-multimodal-concatmodel achieved an accuracy of 92.07% and an F1 score of 91.96%. - Multimodal Synergy: The inclusion of visual data (face and tongue) significantly boosted performance compared to unimodal baselines. For instance, using questionnaire data alone yielded 86.38% accuracy.
- Fusion Strategy: Interestingly, simple feature concatenation was found to be more effective than complex fusion networks, suggesting that the quality of specialized feature extraction is the primary driver of performance.
Conclusion
TCM-CAS represents a significant step toward objective and automated TCM diagnosis. By combining specialized image segmentation with graph-based symptom modeling, the system achieves high accuracy while maintaining clinical interpretability through intermediate feature outputs. Future work aims to expand the system to cover all nine standard TCM constitutions and incorporate additional modalities like voice and pulse data.
Reference: Mao, H., Zhang, B., Xu, H., & Gao, K. (2022). An End-to-End Traditional Chinese Medicine Constitution Assessment System Based on Multimodal Clinical Feature Representation and Fusion. Proceedings of the AAAI Conference on Artificial Intelligence.
Summary updated