TL;DR
SEAT (Sign-Type Evidence-Audited Transport) is a multi-label framework for automated tongue diagnosis that replaces uniform neural heads with specialized pathways for morphology, color, and label co-occurrence. By matching specific clinical evidence types to dedicated prediction and auditing mechanisms, it outperforms much larger vision-language models and ensures that predictions are grounded in clinically relevant features rather than spurious correlations.
What Problem Does This Paper Address?
In Traditional Chinese Medicine (TCM), tongue diagnosis is a non-invasive but highly subjective process. Automating this via computer vision is challenging because tongue signs are "heterogeneous"—they aren't all defined by the same visual cues. For example, "tooth marks" are defined by localized shapes (morphology), while a "pale tongue" is defined by global color (chromaticity).
Most existing deep learning models use a shared backbone and a uniform multi-label head, treating every sign the same way. This creates a "black box" problem: a model might correctly predict a sign based on a spurious correlation (e.g., predicting a crack because of the tongue's color) rather than the actual clinical evidence. Furthermore, standard interpretability tools like Grad-CAM can show where a model looks, but they can't verify why (e.g., if it's responding to redness or texture).
Methods
The authors propose SEAT, which utilizes a frozen DINOv2-Large backbone to extract patch features. Instead of a standard classifier, SEAT employs three distinct pathways:
- Mask-Guided Attention Reader: The core discriminative engine. It uses a pseudo-tongue mask (generated by a frozen Segment Anything Model) to focus sign-specific attention heads on the tongue body, ignoring background noise. This is primarily responsible for detecting morphological signs like cracks and spots.
- C1: Auxiliary Chromatic Transport: A dedicated pathway for the "TonguePale" sign. It operates on raw LAB/HSV color descriptors. It uses Optimal Transport (OT) to match image patches to learnable color prototypes, ensuring the model captures fine-grained chromatic variations that semantic features might miss.
- C2: Co-occurrence Geometry: A regularizer that looks at the relationship between labels. It uses OT to shape the batch-level predicted label distribution based on a "Jaccard cost" derived from training data. This ensures the model respects known clinical patterns (e.g., certain signs often appear together).
Crucially, SEAT includes Evidence Audits. To verify the model, the authors perform intervention tests: suppressing high-weight patches to check morphological reliance and creating "color counterfactuals" (digitally shifting the tongue's color) to ensure the chromatic pathway responds correctly to clinical axes like paleness.
Datasets
The model was evaluated on the TongueDx benchmark, which contains 5,109 images from 4,650 subjects. Each image is annotated with eight non-exclusive binary signs. The authors ensured subject-level disjointness across their training, validation, and test splits to prevent data leakage.
Key Insights and Ideas
- Evidence-Type Alignment: The central "aha moment" is that medical AI shouldn't treat all labels as equal. By architecting the model to look for specific types of evidence (shape vs. color), the authors improve both accuracy and trust.
- Frozen Backbones are Sufficient: SEAT demonstrates that a well-designed "readout" mechanism on top of a strong frozen foundation (DINOv2) can outperform domain-specific models that are fine-tuned end-to-end.
- Optimal Transport as a Regularizer: Using OT to align predicted label marginals with empirical co-occurrence patterns provides a mathematically grounded way to incorporate domain knowledge without forcing rigid per-image constraints.
- Auditing vs. Explaining: The authors argue that post-hoc heatmaps are insufficient for clinical safety. Their "audit" approach—testing the model's causal response to specific feature changes—provides a higher standard of interpretability.
Key Results
- State-of-the-Art Performance: SEAT achieved a mean macro-F1 of 72.59, with a six-model ensemble reaching 74.15.
- Outperforming LLMs: The ensemble outperformed ShizhenGPT-7B-VL (a domain-specialized vision-language model) by 2.99 percentage points, despite SEAT having a much smaller trainable parameter footprint.
- Rare Sign Gains: The specialized pathways led to significant improvements in rare, cue-heavy signs like "TonguePale" and "Ecchymosis," where standard models often struggle due to class imbalance.
- Faithfulness: The audits confirmed that the spatial pathway was truly "load-bearing" for morphological signs and that the color pathway responded specifically to clinical paleness rather than general color shifts.
Limitations and Future Work
- Statistical Scope: The test sets for the rarest signs are quite small (fewer than 100 positive cases), which leads to high variability in performance metrics.
- Generalization: The study was limited to a single dataset (TongueDx). The authors note that cross-site and cross-device generalization (handling different lighting or camera types) remains an open challenge.
- Manual Fusion: Currently, the color pathway is integrated via late fusion with validation-selected weights. Future work could explore more dynamic, input-dependent gating mechanisms.
- Illumination Sensitivity: The color descriptors do not yet account for complex factors like white balance or hue wrap-around, which could affect the robustness of the chromatic pathway.
Summary updated