SEAT: Sign-Type Evidence-Audited Transport for Multi-Label Tongue Diagnosis

Liuyi Xu, Qing Wang, Zikai Liang, An-Yang Lu, Yining QianarXiv 2026 · arXiv

TL;DR

SEAT (Sign-Type Evidence-Audited Transport) is a multi-label framework for automated tongue diagnosis that replaces uniform neural heads with specialized pathways for morphology, color, and label co-occurrence. By matching specific clinical evidence types to dedicated prediction and auditing mechanisms, it outperforms much larger vision-language models and ensures that predictions are grounded in clinically relevant features rather than spurious correlations.

What Problem Does This Paper Address?

In Traditional Chinese Medicine (TCM), tongue diagnosis is a non-invasive but highly subjective process. Automating this via computer vision is challenging because tongue signs are "heterogeneous"—they aren't all defined by the same visual cues. For example, "tooth marks" are defined by localized shapes (morphology), while a "pale tongue" is defined by global color (chromaticity).

Most existing deep learning models use a shared backbone and a uniform multi-label head, treating every sign the same way. This creates a "black box" problem: a model might correctly predict a sign based on a spurious correlation (e.g., predicting a crack because of the tongue's color) rather than the actual clinical evidence. Furthermore, standard interpretability tools like Grad-CAM can show where a model looks, but they can't verify why (e.g., if it's responding to redness or texture).

Methods

The authors propose SEAT, which utilizes a frozen DINOv2-Large backbone to extract patch features. Instead of a standard classifier, SEAT employs three distinct pathways:

  1. Mask-Guided Attention Reader: The core discriminative engine. It uses a pseudo-tongue mask (generated by a frozen Segment Anything Model) to focus sign-specific attention heads on the tongue body, ignoring background noise. This is primarily responsible for detecting morphological signs like cracks and spots.
  2. C1: Auxiliary Chromatic Transport: A dedicated pathway for the "TonguePale" sign. It operates on raw LAB/HSV color descriptors. It uses Optimal Transport (OT) to match image patches to learnable color prototypes, ensuring the model captures fine-grained chromatic variations that semantic features might miss.
  3. C2: Co-occurrence Geometry: A regularizer that looks at the relationship between labels. It uses OT to shape the batch-level predicted label distribution based on a "Jaccard cost" derived from training data. This ensures the model respects known clinical patterns (e.g., certain signs often appear together).

Crucially, SEAT includes Evidence Audits. To verify the model, the authors perform intervention tests: suppressing high-weight patches to check morphological reliance and creating "color counterfactuals" (digitally shifting the tongue's color) to ensure the chromatic pathway responds correctly to clinical axes like paleness.

Datasets

The model was evaluated on the TongueDx benchmark, which contains 5,109 images from 4,650 subjects. Each image is annotated with eight non-exclusive binary signs. The authors ensured subject-level disjointness across their training, validation, and test splits to prevent data leakage.

Key Insights and Ideas

  • Evidence-Type Alignment: The central "aha moment" is that medical AI shouldn't treat all labels as equal. By architecting the model to look for specific types of evidence (shape vs. color), the authors improve both accuracy and trust.
  • Frozen Backbones are Sufficient: SEAT demonstrates that a well-designed "readout" mechanism on top of a strong frozen foundation (DINOv2) can outperform domain-specific models that are fine-tuned end-to-end.
  • Optimal Transport as a Regularizer: Using OT to align predicted label marginals with empirical co-occurrence patterns provides a mathematically grounded way to incorporate domain knowledge without forcing rigid per-image constraints.
  • Auditing vs. Explaining: The authors argue that post-hoc heatmaps are insufficient for clinical safety. Their "audit" approach—testing the model's causal response to specific feature changes—provides a higher standard of interpretability.

Key Results

  • State-of-the-Art Performance: SEAT achieved a mean macro-F1 of 72.59, with a six-model ensemble reaching 74.15.
  • Outperforming LLMs: The ensemble outperformed ShizhenGPT-7B-VL (a domain-specialized vision-language model) by 2.99 percentage points, despite SEAT having a much smaller trainable parameter footprint.
  • Rare Sign Gains: The specialized pathways led to significant improvements in rare, cue-heavy signs like "TonguePale" and "Ecchymosis," where standard models often struggle due to class imbalance.
  • Faithfulness: The audits confirmed that the spatial pathway was truly "load-bearing" for morphological signs and that the color pathway responded specifically to clinical paleness rather than general color shifts.

Limitations and Future Work

  • Statistical Scope: The test sets for the rarest signs are quite small (fewer than 100 positive cases), which leads to high variability in performance metrics.
  • Generalization: The study was limited to a single dataset (TongueDx). The authors note that cross-site and cross-device generalization (handling different lighting or camera types) remains an open challenge.
  • Manual Fusion: Currently, the color pathway is integrated via late fusion with validation-selected weights. Future work could explore more dynamic, input-dependent gating mechanisms.
  • Illumination Sensitivity: The color descriptors do not yet account for complex factors like white balance or hue wrap-around, which could affect the robustness of the chromatic pathway.

Summary updated

Abstract

Automated tongue-sign recognition can improve the consistency of non-invasive examinations that otherwise rely on subjective visual judgment. A single tongue image often exhibits multiple coexisting signs characterized by heterogeneous evidence, including localized morphology, global chromaticity, and inter-label dependence. However, existing methods typically rely on shared representations and uniform multi-label heads, making it difficult to verify whether predictions stem from clinically relevant evidence rather than spurious visual correlations. We propose SEAT, an evidence-audited framework that matches distinct evidence types with dedicated prediction and evaluation mechanisms. Built on frozen DINOv2 patch features, SEAT couples mask-guided sign-specific attention for localized morphology, a semi-relaxed optimal transport pathway over LAB/HSV color prototypes to enhance pale-tongue recognition, and a co-occurrence-aware transport regularizer to shape the batch-level predicted label marginal. These components are evaluated through audits matched to their evidence types. On the TongueDx benchmark, SEAT achieves a mean macro-F1 of 72.59 with a standard deviation of 1.04 across six random seeds, with a six-model ensemble reaching 74.15. The ensemble outperforms a backbone-matched baseline and a larger, domain-specialized vision-language model by 8.06 and 2.99 percentage points, respectively. Our results demonstrate that aligning dedicated prediction pathways and audits with sign-specific characteristics improves multi-label accuracy while offering more targeted evidential grounding than uniform post-hoc spatial explanations. Code is available at https://github.com/NEU-TCM-Lab/Sign-type-Evidence-Audited-Transport.
BibTeX
@misc{xu2026seat,
  author = {Liuyi Xu and Qing Wang and Zikai Liang and An-Yang Lu and Yining Qian},
  title = {{SEAT: Sign-Type Evidence-Audited Transport for Multi-Label Tongue Diagnosis}},
  year = {2026},
  eprint = {2611.19748},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19748},
}
Figure