Abstract: Tongue diagnosis is a fundamental diagnostic method in Traditional Chinese Medicine (TCM), yet its reliance on practitioner experience leads to subjectivity and inconsistency. We present WSVM, a fully automated framework for tongue extraction and tooth-marked tongue recognition. By leveraging Vision Transformers (ViT) and Multiple Instance Learning (MIL), WSVM identifies tooth-marked regions using only image-level annotations, bypassing the need for labor-intensive manual bounding boxes. Our approach achieves high diagnostic accuracy and provides interpretable visualizations of clinical features, offering significant value for objective TCM diagnosis.
Introduction
In Traditional Chinese Medicine (TCM), the tongue serves as a vital indicator of internal health. Specifically, a "tooth-marked tongue"—one featuring indentations along its lateral edges—often signals underlying physiological imbalances. While automated image analysis offers a path toward objective diagnosis, traditional deep learning models for object detection require thousands of meticulously annotated images. Manually drawing bounding boxes around every indentation is a significant bottleneck for clinical experts, as it requires pixel-perfect precision across massive datasets.
The WSVM Framework
To address the annotation bottleneck, we developed the Weakly Supervised method using Vision transformer and Multiple instance learning (WSVM). This two-stage framework automates the entire pipeline from raw clinical photograph to diagnostic output.
Stage 1: Automated Tongue Extraction
The first challenge is isolating the tongue from distracting background elements like lips, teeth, and skin. We employ a combination of YOLOv8 for initial bounding box detection and the Segment Anything Model (SAM) for precise mask generation. By using the YOLOv8 box as a "prompt" for SAM, the system achieves highly stable and accurate segmentation, ensuring that the subsequent recognition model focuses solely on relevant tongue tissue without interference from surrounding facial features.
Stage 2: Weakly Supervised Recognition
The core innovation of WSVM lies in its use of Vision Transformers (ViT) for Weakly Supervised Object Detection (WSOD). Instead of requiring experts to label every tooth mark, the model is trained using only image-level labels (e.g., "tooth-marked" vs. "normal").
- ViT as a Natural Proposer: ViT naturally divides the tongue image into a grid of patches. In our framework, each patch is treated as an "instance" within a "bag" (the whole image). This eliminates the need for complex external region proposal algorithms.
- Multiple Instance Learning (MIL): The Multiple Instance Calculation Module (MICM) evaluates these patches. Following MIL logic, if the model identifies at least one patch as containing a tooth mark, the entire image is classified as tooth-marked.
- Handling Class Imbalance: Because tooth marks are often small and sparse compared to the rest of the tongue surface, we utilize a combination of Cross-Entropy and Focal Loss to prevent the model from being overwhelmed by the majority of "normal" tongue tissue.
Experimental Results
We evaluated WSVM on a private clinical dataset of 1,745 images from the Third Affiliated Hospital of Fujian University of TCM and a public Kaggle dataset. The tongue extraction stage achieved near-perfect performance with an mAP50 of 0.995. For tooth-marked recognition, WSVM reached an accuracy of 84.2% on the private dataset and 89.0% on the public dataset, outperforming existing state-of-the-art weakly supervised methods like MILCNN and WSTDN by up to 5 percentage points.
Beyond raw accuracy, visualization experiments confirm that the model's attention is correctly focused on the tongue's edges where indentations occur. This interpretability is crucial for clinical adoption, as it allows practitioners to verify the model's reasoning and ensures the diagnosis is based on relevant clinical features rather than artifacts.
Conclusion
WSVM provides a robust, objective, and interpretable tool for TCM tongue diagnosis. By reducing the reliance on expert-level manual annotations, this framework paves the way for broader automated diagnostic tools in specialized medical fields. Future work will expand the system to recognize additional features, such as tongue coating thickness and color, further enhancing its clinical utility for comprehensive health assessment.
Summary updated