TL;DR
OmniTongue is the first comprehensive, fine-grained benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on Traditional Chinese Medicine (TCM) tongue diagnosis. By introducing 16 clinical dimensions and region-specific annotations, the researchers reveal a "capability illusion" where current models rely on statistical shortcuts rather than a genuine understanding of tongue pathology.
What Problem Does This Paper Address?
While MLLMs have shown impressive results in Western medical imaging (like X-rays or MRIs), their application in Traditional Chinese Medicine (TCM) remains largely untested. Tongue diagnosis is a cornerstone of TCM, requiring a practitioner to observe both global features (like overall tongue shape) and localized details (like coating thickness at the root vs. the tip).
Existing datasets for this task are often too simplistic, providing only single global labels or coarse categories. This lack of granularity makes it impossible to assess whether an AI truly "understands" the spatial and pathological nuances of a tongue or is just making lucky guesses. The authors address this by creating a rigorous, expert-validated Visual Question Answering (VQA) benchmark that forces models to perform fine-grained, region-conditioned recognition.
Methods
The researchers developed OmniTongue by transforming expert clinical annotations into a VQA format.
- Expert Annotation: Two senior TCM physicians (10+ years experience) independently annotated images across 16 dimensions. A third expert adjudicated any disagreements.
- Multidimensional Design: The benchmark covers three main groupings:
- Global Tongue Body: Color, size, shape, teeth marks, cracks, and petechiae (spots).
- Global Tongue Coating: Overall color, thickness, distribution, and texture.
- Regional Tongue Coating: Color and thickness specifically localized to the Tip (Heart-Lung), Middle (Spleen-Stomach), and Root (Kidney).
- VQA Generation: Labels were converted into 21,616 multiple-choice question-answer pairs. To prevent models from simply memorizing phrasing, they used LLMs to paraphrase questions into diverse natural language expressions while keeping the medical meaning intact.
- Evaluation Protocol: They tested nine mainstream MLLMs (including Qwen, InternVL, and the medical-specific HuatuoGPT) using Macro-F1 as the primary metric to account for the highly imbalanced nature of clinical data.
Datasets
The authors introduce the OmniTongue dataset, which consists of:
- 1,351 high-resolution images collected from three clinical sites.
- 21,616 instruction pairs (VQA items).
- 66 fine-grained labels (e.g., distinguishing between "light red," "crimson," and "dark purple").
The dataset features a "long-tailed" distribution, meaning common conditions are frequent, while critical pathological signs are rare—a realistic reflection of clinical practice that poses a major challenge for AI.
Key Insights and Ideas
- The Capability Illusion: The most striking finding is that high Top-1 Accuracy is misleading. Models often achieve high accuracy by simply predicting the most common class (e.g., "Normal"). However, their Macro-F1 scores—which value performance on rare classes—are often dismal. For example, Qwen3-VL hits 90.3% accuracy on "Spots" but its Macro-F1 collapses to 29.0%.
- The Localization Bottleneck: Current MLLMs struggle with "spatial grounding." They might correctly identify a feature but fail to place it in the correct anatomical region (tip vs. root). Standard vision-language alignment isn't enough for the precision required in medical diagnostics.
- Medical Specialization vs. General Scaling: Surprisingly, medical-specific models like HuatuoGPT-Vision did not show a significant advantage in regional tasks. This suggests that pre-training on medical text/images without explicit spatial grounding doesn't automatically translate to better localized diagnostic ability.
Key Results
The evaluation showed that even the strongest models have a long way to go:
- Global Body Features: Qwen3-VL led with an average Macro-F1 of 35.2%, significantly outperforming the traditional BLIP-2 baseline (18.0%).
- Global Coating Features: HuatuoGPT achieved the highest average Macro-F1 (40.3%), though it still struggled with specific categories like coating distribution.
- Regional Performance: This was the hardest category. Qwen3-VL achieved an average Macro-F1 of 38.1% across regional coating features, but performance dropped significantly for specific areas like "Tip Coating Color" (29.8%).

Limitations and Future Work
The authors acknowledge several areas for growth:
- Metadata: The current release lacks patient age and sex metadata, which are important context for TCM diagnosis.
- Model Scope: The study focused on nine open-source models; future work will include closed-source models (like GPT-4o) and more specialized medical MLLMs.
- Anatomy Awareness: The results highlight a desperate need for "anatomy-aware" models that can explicitly ground medical concepts to specific physical regions.
OmniTongue sets a new, higher bar for AI in TCM, moving the field away from simple classification and toward authentic, expert-level clinical reasoning.
Summary updated