TL;DR
Researchers have developed WSVM, a two-stage automated framework that identifies "tooth-marked tongues"—a key diagnostic sign in Traditional Chinese Medicine (TCM)—using only image-level labels. By combining the patch-based architecture of Vision Transformers (ViT) with Multiple Instance Learning (MIL), the system can pinpoint specific regions of interest without requiring the tedious, pixel-perfect manual annotations usually needed for object detection.
What Problem Does This Paper Address?
In Traditional Chinese Medicine, the appearance of the tongue is a window into a patient's health. A "tooth-marked tongue" (showing indentations along the edges caused by pressure from teeth) often indicates specific underlying conditions. However, traditional diagnosis is highly subjective, relying on a practitioner's experience and eyesight.
While deep learning offers a path toward objective, automated diagnosis, it faces a massive bottleneck: data annotation. Training a standard object detector to find tooth marks requires experts to manually draw boxes or masks around every single indentation across thousands of images. This paper tackles this by using Weakly Supervised Object Detection (WSOD). The goal is to train a model that can "see" where the tooth marks are, even though it was only told whether an entire image was "tooth-marked" or "normal."
Methods
The authors propose a two-stage pipeline called WSVM (Weakly Supervised method using Vision transformer and Multiple instance learning):
- Stage 1: Automated Tongue Extraction: To remove distracting backgrounds (lips, teeth, skin), the team uses YOLOv8 to detect the tongue's bounding box and the Segment Anything Model (SAM) to generate a precise mask. They found that using a bounding box as a "prompt" for SAM yielded much more stable results than using simple point prompts.
- Stage 2: Weakly Supervised Recognition:
- The extracted tongue image is fed into a Vision Transformer (ViT).
- The ViT naturally divides the image into small patches. In this framework, these patches are treated as "instances" within a "bag" (the whole image).
- A Multiple Instance Calculation Module (MICM) processes these patches. Following the logic of Multiple Instance Learning (MIL), if at least one patch is identified as "positive" (containing a tooth mark), the entire image is classified as tooth-marked.
- The model uses a combination of Cross-Entropy and Focal Loss to handle the inherent class imbalance (as tooth marks are often small and sparse compared to the rest of the tongue).
Datasets
The study utilized two primary sources:
- Private Dataset: 1,745 high-resolution clinical images collected from the Third Affiliated Hospital of Fujian University of TCM. This included 556 tooth-marked and 1,188 non-tooth-marked samples, annotated by two doctors with over 10 years of experience.
- Public Dataset: A tooth-marked tongue dataset from Kaggle, consisting of 546 marked and 704 unmarked images, used to verify the model's generalizability.
Key Insights and Ideas
- ViT as a Natural Proposer: Standard weakly supervised methods often use complex "region proposal" algorithms to guess where objects might be. This paper realizes that ViT’s inherent patch-based processing is a region proposal mechanism. Each patch is a candidate region, making the architecture perfectly suited for MIL.
- The Power of Box Prompts: The authors observed that point-based prompts for SAM often failed on "scalloped" tongues, accidentally including teeth in the segmentation. Switching to YOLOv8-generated bounding boxes as prompts provided the "global" context SAM needed for a clean crop.
- Interpretability via Attention: By visualizing the model’s attention and the MICM outputs, the researchers proved the model wasn't just "guessing" based on color; it was actually looking at the indentations on the tongue's edges to make its decision.
Key Results
- Tongue Extraction: The YOLOv8 + SAM stage achieved a near-perfect 1.0 accuracy and recall, with an mAP50 of 0.995, ensuring the second stage always had a clean image to work with.
- Recognition Accuracy: WSVM achieved an average accuracy of 84.2% on the private dataset and 89.0% on the public Kaggle dataset.
- Comparison: WSVM outperformed existing state-of-the-art methods (like MILCNN and WSTDN) by roughly 2.8 to 5 percentage points in accuracy. It also showed significantly higher precision and F1 scores, proving it is more robust at distinguishing subtle tooth marks from normal tongue edges.
Limitations and Future Work
While the model performs well, the authors note that it currently focuses on tooth marks and "cracked" tongues (tested in their ablation studies). Future work aims to extend this framework to other TCM diagnostic features, such as tongue coating color and thickness. Additionally, while the model reduces the need for pixel-level labels, it still requires a significant amount of image-level labeled data, which can still be a hurdle in specialized medical fields.
Summary updated