OmniTongue: A Comprehensive MLLM Benchmark for Automated Tongue Diagnosis in Traditional Chinese Medicine

Liuyi Xu, Yun Guo, Shusheng Li, Xuewen Zhang, Yining Qian, An-Yang LuarXiv 2026 · arXiv

TL;DR

OmniTongue is the first comprehensive, fine-grained benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on Traditional Chinese Medicine (TCM) tongue diagnosis. By introducing 16 clinical dimensions and region-specific annotations, the researchers reveal a "capability illusion" where current models rely on statistical shortcuts rather than a genuine understanding of tongue pathology.

What Problem Does This Paper Address?

While MLLMs have shown impressive results in Western medical imaging (like X-rays or MRIs), their application in Traditional Chinese Medicine (TCM) remains largely untested. Tongue diagnosis is a cornerstone of TCM, requiring a practitioner to observe both global features (like overall tongue shape) and localized details (like coating thickness at the root vs. the tip).

Existing datasets for this task are often too simplistic, providing only single global labels or coarse categories. This lack of granularity makes it impossible to assess whether an AI truly "understands" the spatial and pathological nuances of a tongue or is just making lucky guesses. The authors address this by creating a rigorous, expert-validated Visual Question Answering (VQA) benchmark that forces models to perform fine-grained, region-conditioned recognition.

Methods

The researchers developed OmniTongue by transforming expert clinical annotations into a VQA format.

  1. Expert Annotation: Two senior TCM physicians (10+ years experience) independently annotated images across 16 dimensions. A third expert adjudicated any disagreements.
  2. Multidimensional Design: The benchmark covers three main groupings:
  3. Global Tongue Body: Color, size, shape, teeth marks, cracks, and petechiae (spots).
  4. Global Tongue Coating: Overall color, thickness, distribution, and texture.
  5. Regional Tongue Coating: Color and thickness specifically localized to the Tip (Heart-Lung), Middle (Spleen-Stomach), and Root (Kidney).
  6. VQA Generation: Labels were converted into 21,616 multiple-choice question-answer pairs. To prevent models from simply memorizing phrasing, they used LLMs to paraphrase questions into diverse natural language expressions while keeping the medical meaning intact.
  7. Evaluation Protocol: They tested nine mainstream MLLMs (including Qwen, InternVL, and the medical-specific HuatuoGPT) using Macro-F1 as the primary metric to account for the highly imbalanced nature of clinical data.

Datasets

The authors introduce the OmniTongue dataset, which consists of:

  • 1,351 high-resolution images collected from three clinical sites.
  • 21,616 instruction pairs (VQA items).
  • 66 fine-grained labels (e.g., distinguishing between "light red," "crimson," and "dark purple").
Fig. 1. Overview of our OmniTongue dataset. OmniTongue covers the majority of critical diagnostic features and regional characteristics of the human tongue, such as tongue size, shape, color, fissure conditions, teeth marks, as well as coating attributes (thickness, color, distribution, and texture) across the tip, middle, and root areas.

The dataset features a "long-tailed" distribution, meaning common conditions are frequent, while critical pathological signs are rare—a realistic reflection of clinical practice that poses a major challenge for AI.

Fig. 2. Dataset VQA: Distribution of 16 comprehensive clinical dimensions. Colored blocks within each column denote distinct categories, stacked bottom-up by decreasing sample frequency. OmniTongue exhibits a pronounced long-tailed distribu- tion mirroring authentic clinical scenarios, where common conditions dominate over critical but rare pathological signs. This natural imbalance challenges MLLMs to rec- ognize minority-class local lesions instead of exploiting global statistical shortcuts.

Key Insights and Ideas

  • The Capability Illusion: The most striking finding is that high Top-1 Accuracy is misleading. Models often achieve high accuracy by simply predicting the most common class (e.g., "Normal"). However, their Macro-F1 scores—which value performance on rare classes—are often dismal. For example, Qwen3-VL hits 90.3% accuracy on "Spots" but its Macro-F1 collapses to 29.0%.
  • The Localization Bottleneck: Current MLLMs struggle with "spatial grounding." They might correctly identify a feature but fail to place it in the correct anatomical region (tip vs. root). Standard vision-language alignment isn't enough for the precision required in medical diagnostics.
  • Medical Specialization vs. General Scaling: Surprisingly, medical-specific models like HuatuoGPT-Vision did not show a significant advantage in regional tasks. This suggests that pre-training on medical text/images without explicit spatial grounding doesn't automatically translate to better localized diagnostic ability.

Key Results

The evaluation showed that even the strongest models have a long way to go:

  • Global Body Features: Qwen3-VL led with an average Macro-F1 of 35.2%, significantly outperforming the traditional BLIP-2 baseline (18.0%).
  • Global Coating Features: HuatuoGPT achieved the highest average Macro-F1 (40.3%), though it still struggled with specific categories like coating distribution.
  • Regional Performance: This was the hardest category. Qwen3-VL achieved an average Macro-F1 of 38.1% across regional coating features, but performance dropped significantly for specific areas like "Tip Coating Color" (29.8%).

Table 1: Benchmark results for Global Tongue Body Features. Table 3: Benchmark results for Regional Coating Features, showing the difficulty of localized diagnosis.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Metadata: The current release lacks patient age and sex metadata, which are important context for TCM diagnosis.
  • Model Scope: The study focused on nine open-source models; future work will include closed-source models (like GPT-4o) and more specialized medical MLLMs.
  • Anatomy Awareness: The results highlight a desperate need for "anatomy-aware" models that can explicitly ground medical concepts to specific physical regions.

OmniTongue sets a new, higher bar for AI in TCM, moving the field away from simple classification and toward authentic, expert-level clinical reasoning.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have excelled in Western medical imaging, yet their potential in Traditional Chinese Medicine (TCM) tongue diagnosis remains systematically unevaluated. Tongue diagnosis requires sophisticated global and local perception, serving as a rigorous touchstone for assessing the visual understanding depth of MLLMs. However, existing datasets are limited by single dimensions, coarse granularity, and ambiguous labels, hindering the evaluation of authentic clinical capabilities. To bridge this gap, we introduce OmniTongue, the first comprehensive, fine-grained Visual Question Answering benchmark for TCM tongue diagnosis. This benchmark integrates 1,351 expert-validated images and 21,616 instruction pairs. Its core design features 16 clinical dimensions for multidimensional evaluation, region-specific annotations for pathological localization, and 66 fine-grained labels for precise diagnostic representation. Extensive evaluations of existing MLLMs reveal significant performance gaps, demonstrating that medical-specific models lack domain advantages, while universally poor Macro-F1 scores expose a reliance on statistical shortcuts rather than genuine understanding of tongue pathology. These findings not only highlight the current limitations of MLLMs in interpreting regional medical features but also underscore OmniTongue's significance for developing robust automated tongue diagnosis models. The code and dataset will be made publicly available in a future release.
BibTeX
@misc{xu2026omnitongue,
  author = {Liuyi Xu and Yun Guo and Shusheng Li and Xuewen Zhang and Yining Qian and An-Yang Lu},
  title = {{OmniTongue: A Comprehensive MLLM Benchmark for Automated Tongue Diagnosis in Traditional Chinese Medicine}},
  year = {2026},
  eprint = {2611.22652},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22652},
}
Figure