Weakly Supervised Object Detection for Automatic Tooth-marked Tongue Recognition

Yongcun Zhang, Jiajun Xu, Yina He, Shaozi Li, Zhiming Luo, Huangwei Lei

Abstract: Tongue diagnosis is a fundamental diagnostic method in Traditional Chinese Medicine (TCM), yet its reliance on practitioner experience leads to subjectivity and inconsistency. We present WSVM, a fully automated framework for tongue extraction and tooth-marked tongue recognition. By leveraging Vision Transformers (ViT) and Multiple Instance Learning (MIL), WSVM identifies tooth-marked regions using only image-level annotations, bypassing the need for labor-intensive manual bounding boxes. Our approach achieves high diagnostic accuracy and provides interpretable visualizations of clinical features, offering significant value for objective TCM diagnosis.

Introduction

In Traditional Chinese Medicine (TCM), the tongue serves as a vital indicator of internal health. Specifically, a "tooth-marked tongue"—one featuring indentations along its lateral edges—often signals underlying physiological imbalances. While automated image analysis offers a path toward objective diagnosis, traditional deep learning models for object detection require thousands of meticulously annotated images. Manually drawing bounding boxes around every indentation is a significant bottleneck for clinical experts, as it requires pixel-perfect precision across massive datasets.

The WSVM Framework

To address the annotation bottleneck, we developed the Weakly Supervised method using Vision transformer and Multiple instance learning (WSVM). This two-stage framework automates the entire pipeline from raw clinical photograph to diagnostic output.

The WSVM framework consists of a first stage for automated tongue extraction and a second stage for weakly supervised recognition.

Stage 1: Automated Tongue Extraction

The first challenge is isolating the tongue from distracting background elements like lips, teeth, and skin. We employ a combination of YOLOv8 for initial bounding box detection and the Segment Anything Model (SAM) for precise mask generation. By using the YOLOv8 box as a "prompt" for SAM, the system achieves highly stable and accurate segmentation, ensuring that the subsequent recognition model focuses solely on relevant tongue tissue without interference from surrounding facial features.

Stage 2: Weakly Supervised Recognition

The core innovation of WSVM lies in its use of Vision Transformers (ViT) for Weakly Supervised Object Detection (WSOD). Instead of requiring experts to label every tooth mark, the model is trained using only image-level labels (e.g., "tooth-marked" vs. "normal").

  • ViT as a Natural Proposer: ViT naturally divides the tongue image into a grid of patches. In our framework, each patch is treated as an "instance" within a "bag" (the whole image). This eliminates the need for complex external region proposal algorithms.
  • Multiple Instance Learning (MIL): The Multiple Instance Calculation Module (MICM) evaluates these patches. Following MIL logic, if the model identifies at least one patch as containing a tooth mark, the entire image is classified as tooth-marked.
  • Handling Class Imbalance: Because tooth marks are often small and sparse compared to the rest of the tongue surface, we utilize a combination of Cross-Entropy and Focal Loss to prevent the model from being overwhelmed by the majority of "normal" tongue tissue.

Experimental Results

We evaluated WSVM on a private clinical dataset of 1,745 images from the Third Affiliated Hospital of Fujian University of TCM and a public Kaggle dataset. The tongue extraction stage achieved near-perfect performance with an mAP50 of 0.995. For tooth-marked recognition, WSVM reached an accuracy of 84.2% on the private dataset and 89.0% on the public dataset, outperforming existing state-of-the-art weakly supervised methods like MILCNN and WSTDN by up to 5 percentage points.

Visualization of tooth-marked tongue recognition. (b-d) show expert ground truth, while (f-g) demonstrate WSVM's ability to localize indentations compared to other methods (e).

Beyond raw accuracy, visualization experiments confirm that the model's attention is correctly focused on the tongue's edges where indentations occur. This interpretability is crucial for clinical adoption, as it allows practitioners to verify the model's reasoning and ensures the diagnosis is based on relevant clinical features rather than artifacts.

Conclusion

WSVM provides a robust, objective, and interpretable tool for TCM tongue diagnosis. By reducing the reliance on expert-level manual annotations, this framework paves the way for broader automated diagnostic tools in specialized medical fields. Future work will expand the system to recognize additional features, such as tongue coating thickness and color, further enhancing its clinical utility for comprehensive health assessment.

Summary updated

Abstract

Tongue diagnosis in Traditional Chinese Medicine (TCM) is a crucial diagnostic method that can reflect an individual's health status. Traditional methods for identifying tooth-marked tongues are subjective and inconsistent because they rely on practitioner experience. We propose a novel fully automated Weakly Supervised method using Vision transformer and Multiple instance learning WSVM for tongue extraction and tooth-marked tongue recognition. Our approach first accurately detects and extracts the tongue region from clinical images, removing any irrelevant background information. Then, we implement an end-to-end weakly supervised object detection method. We utilize Vision Transformer (ViT) to process tongue images in patches and employ multiple instance loss to identify tooth-marked regions with only image-level annotations. WSVM achieves high accuracy in tooth-marked tongue classification, and visualization experiments demonstrate its effectiveness in pinpointing these regions. This automated approach enhances the objectivity and accuracy of tooth-marked tongue diagnosis. It provides significant clinical value by assisting TCM practitioners in making precise diagnoses and treatment recommendations. Code is available at https://github.com/yc-zh/WSVM.
BibTeX
@article{zhang2025weakly,
  author = {Yongcun Zhang and Jiajun Xu and Yina He and Shaozi Li and Zhiming Luo and Huangwei Lei},
  title = {{Weakly Supervised Object Detection for Automatic Tooth-marked Tongue Recognition}},
  journal = {Biomedical Signal Processing and Control},
  year = {2025},
  url = {https://openreview.net/forum?id=ULlzcw5XvF},
}
Figure