TL;DR
K-MaT (Knowledge-Anchored Manifold Transport) is a novel prompt-learning framework designed to help medical AI models generalize from high-end imaging (like CT or Mammography) to low-end, frontline modalities (like X-ray or Ultrasound). By using LLM-generated clinical anchors and optimal transport to align data structures, K-MaT achieves state-of-the-art zero-shot performance on low-end modalities without requiring a single training image from those domains.
What Problem Does This Paper Address?
In medical imaging, there is a massive "modality gap." A model trained to detect pneumonia on high-resolution CT scans often fails miserably when asked to do the same on a standard chest X-ray. This happens because deep learning models tend to learn "modality-specific shortcuts"—features related to the physics of the imaging equipment rather than the underlying pathology.
While Vision-Language Models (VLMs) like BiomedCLIP offer a way to adapt to new tasks via prompt tuning, they suffer from catastrophic knowledge forgetting in cross-modal settings. When you tune a prompt for a high-end modality, the model "forgets" the general medical semantics, causing performance to collapse when deployed on a different, lower-end modality. The authors sought a way to transfer the "decision structure" learned from high-end data to low-end data in a strictly zero-shot regime.
Methods
K-MaT introduces three primary innovations to the standard prompt-tuning pipeline (built on a frozen BiomedCLIP backbone):
- Factorized Prompt Parameterization: Instead of a single prompt, K-MaT uses a factorized approach that separates Class-Specific Context (CSC) and Modality-Specific Context (MSC). This allows the model to learn distinct nuances for different imaging types while maintaining shared class meanings.
- Space Anchoring ($L_{anc}$): To prevent the learnable prompts from drifting into "modality-specific nonsense," the authors used GPT-5 to generate 50 clinical descriptions for every class and modality. These descriptions act as a "semantic north star." The learnable prompts are regularized to stay close to these fixed, LLM-generated clinical prototypes.
- Manifold Alignment via Fused Gromov-Wasserstein (FGW) Transport: This is the core technical contribution. K-MaT uses FGW optimal transport to ensure that the relational structure of the low-end prompt manifold mirrors the high-end one. Essentially, if "Malignant" and "Benign" are separated by a certain distance and structure in the CT space, the model forces the X-ray space to maintain that same logical relationship.
Datasets
The authors evaluated K-MaT on four diverse cross-modal pairs:
- Skin I & II: Transferring from Dermoscopy (high-end) to Clinical photos (low-end) using the MRA-MIDAS and Derm7pt datasets.
- Chest: Transferring from CT (CC-CCII) to Chest X-ray.
- Breast: Transferring from Mammography (DMID) to Ultrasound (BUSI).
In all cases, the model was trained using a 16-shot setting on the high-end modality and tested zero-shot on the low-end modality.
Key Insights and Ideas
- Structure Over Statistics: The most important "aha moment" is that we shouldn't just try to match features across modalities; we should match the relationships between classes. By preserving the manifold structure, the model retains the diagnostic logic even when the visual input changes drastically.
- LLMs as Clinical Anchors: Using an LLM to generate a large ensemble of descriptions provides a robust "prior" that prevents the model from overfitting to the visual quirks of a specific dataset.
- The "Harmonic Mean" Metric: The paper highlights the Harmonic Mean (H) of accuracy between modalities. This is a more honest metric for cross-modal transfer because it penalizes models that perform excellently on the source (high-end) but fail on the target (low-end), encouraging balanced generalization.
Key Results
K-MaT consistently outperformed state-of-the-art baselines like CoOp and BiomedCoOp:
- Overall Performance: Achieved an average Harmonic Mean accuracy of 44.1% (up from BiomedCoOp’s 42.0%) and a macro-F1 of 36.2%.
- Preventing Collapse: On the challenging breast imaging task, standard CoOp plummeted to 27.0% accuracy on the low-end (Ultrasound). K-MaT maintained a much more robust 38.4% on the low-end while keeping high-end accuracy at 73.1%.
- Ablation Success: The ablation studies showed that the combination of MSC/CSC and the FGW alignment provided a 10.10% boost in Harmonic Mean accuracy over the baseline, proving that structural alignment is the key to cross-modal stability.
Limitations and Future Work
The authors acknowledge two main hurdles:
- Absolute Performance: While K-MaT is better than existing methods, the absolute accuracy on low-end modalities is still relatively low compared to supervised models trained directly on that data.
- Visual-Textual Gaps: In cases where the visual discrepancy between modalities is extreme (e.g., very different physics), purely text-anchored alignment might not be enough to bridge the gap.
Future Work: The team plans to explore incorporating more reliable visual signals (perhaps through self-supervised pre-training) to further stabilize the low-end transfer capabilities.
Summary updated