An End-to-End Traditional Chinese Medicine Constitution Assessment System Based on Multimodal Clinical Feature Representation and Fusion

Huisheng Mao, Baozheng Zhang, Hua Xu, Kai Gao

Introduction

Traditional Chinese Medicine (TCM) relies heavily on the concept of "Constitution" (TCMC), which characterizes an individual's physiological state and susceptibility to diseases. Traditionally, TCMC is assessed through four diagnostic methods: inspection, listening/smelling, inquiry, and pulse-taking. Automating this process presents significant challenges, primarily in capturing nuanced clinical features (like tongue luster or facial color) and effectively fusing disparate data sources such as visual imagery and structured questionnaire data.

The TCM-CAS Architecture

To address these challenges, researchers presented TCM-CAS at AAAI-22. This end-to-end system automates TCMC assessment by integrating multimodal deep learning techniques. The architecture is divided into four primary modules: Face, Tongue-Dorsal, Question, and Prediction.

Overview of the TCM-CAS architecture, showing the processing pipeline for face, tongue, and questionnaire data.

The Face Module utilizes the Mediapipe framework for landmark detection and alignment, followed by segmentation to extract TCM-relevant features like lip and facial color. Similarly, the Tongue-Dorsal Module employs the MiniSeg model for precise tongue segmentation, focusing on tongue body and coating characteristics.

Graph-Based Symptom Modeling

The most innovative component of TCM-CAS is the Question Module. Rather than treating questionnaire responses as isolated text strings, the system constructs a heterogeneous graph where nodes represent patients and symptoms. Edges in the graph denote the co-occurrence probability of symptoms, allowing the model to capture complex, holistic relationships between physical indicators.

Heterogeneous graph representation used to model relationships between patients (P) and symptoms (Q).

By applying GraphSAGE, the system learns rich embeddings for these nodes. This approach mimics the reasoning of human TCM practitioners who look for patterns of symptoms rather than individual data points.

Results and Performance

The system was evaluated on a dataset of 5,515 samples across four TCMC categories. The results demonstrate the superiority of the multimodal approach:

  • State-of-the-Art Accuracy: The GraphSAGE-multimodal-concat model achieved an accuracy of 92.07% and an F1 score of 91.96%.
  • Multimodal Synergy: The inclusion of visual data (face and tongue) significantly boosted performance compared to unimodal baselines. For instance, using questionnaire data alone yielded 86.38% accuracy.
  • Fusion Strategy: Interestingly, simple feature concatenation was found to be more effective than complex fusion networks, suggesting that the quality of specialized feature extraction is the primary driver of performance.

Conclusion

TCM-CAS represents a significant step toward objective and automated TCM diagnosis. By combining specialized image segmentation with graph-based symptom modeling, the system achieves high accuracy while maintaining clinical interpretability through intermediate feature outputs. Future work aims to expand the system to cover all nine standard TCM constitutions and incorporate additional modalities like voice and pulse data.

Reference: Mao, H., Zhang, B., Xu, H., & Gao, K. (2022). An End-to-End Traditional Chinese Medicine Constitution Assessment System Based on Multimodal Clinical Feature Representation and Fusion. Proceedings of the AAAI Conference on Artificial Intelligence.

Summary updated

Abstract

Traditional Chinese Medicine (TCM) constitution is a fundamental concept in TCM theory. It is determined by multimodal TCM clinical features which, in turn, are obtained from TCM clinical information of image (face, tongue, etc.), audio (pulse and voice), and text (inquiry) modality. The auto assessment of TCM constitution is faced with two major challenges: (1) learning discriminative TCM clinical feature representations; (2) jointly processing the features using multimodal fusion techniques. The TCM Constitution Assessment System (TCM-CAS) is proposed to provide an end-to-end solution to this task, along with auxiliary functions to aid TCM researchers. To improve the results of TCM constitution prediction, the system combines multiple machine learning algorithms such as facial landmark detection, image segmentation, graph neural networks and multimodal fusion. Extensive experiments are conducted on a four-category multimodal TCM constitution dataset, and the proposed method achieves state-of-the-art accuracy. Provided with datasets containing annotations of diseases, the system can also perform automatic disease diagnosis from a TCM perspective.
BibTeX
@inproceedings{mao2022end,
  author = {Huisheng Mao and Baozheng Zhang and Hua Xu and Kai Gao},
  title = {{An End-to-End Traditional Chinese Medicine Constitution Assessment System Based on Multimodal Clinical Feature Representation and Fusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21727},
}
Figure