K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging

Jiajun Zeng, Shadi Albarqouni

TL;DR

K-MaT (Knowledge-Anchored Manifold Transport) is a novel prompt-learning framework designed to help medical AI models generalize from high-end imaging (like CT or Mammography) to low-end, frontline modalities (like X-ray or Ultrasound). By using LLM-generated clinical anchors and optimal transport to align data structures, K-MaT achieves state-of-the-art zero-shot performance on low-end modalities without requiring a single training image from those domains.

What Problem Does This Paper Address?

In medical imaging, there is a massive "modality gap." A model trained to detect pneumonia on high-resolution CT scans often fails miserably when asked to do the same on a standard chest X-ray. This happens because deep learning models tend to learn "modality-specific shortcuts"—features related to the physics of the imaging equipment rather than the underlying pathology.

While Vision-Language Models (VLMs) like BiomedCLIP offer a way to adapt to new tasks via prompt tuning, they suffer from catastrophic knowledge forgetting in cross-modal settings. When you tune a prompt for a high-end modality, the model "forgets" the general medical semantics, causing performance to collapse when deployed on a different, lower-end modality. The authors sought a way to transfer the "decision structure" learned from high-end data to low-end data in a strictly zero-shot regime.

Methods

K-MaT introduces three primary innovations to the standard prompt-tuning pipeline (built on a frozen BiomedCLIP backbone):

  1. Factorized Prompt Parameterization: Instead of a single prompt, K-MaT uses a factorized approach that separates Class-Specific Context (CSC) and Modality-Specific Context (MSC). This allows the model to learn distinct nuances for different imaging types while maintaining shared class meanings.
  2. Space Anchoring ($L_{anc}$): To prevent the learnable prompts from drifting into "modality-specific nonsense," the authors used GPT-5 to generate 50 clinical descriptions for every class and modality. These descriptions act as a "semantic north star." The learnable prompts are regularized to stay close to these fixed, LLM-generated clinical prototypes.
  3. Manifold Alignment via Fused Gromov-Wasserstein (FGW) Transport: This is the core technical contribution. K-MaT uses FGW optimal transport to ensure that the relational structure of the low-end prompt manifold mirrors the high-end one. Essentially, if "Malignant" and "Benign" are separated by a certain distance and structure in the CT space, the model forces the X-ray space to maintain that same logical relationship.

Datasets

The authors evaluated K-MaT on four diverse cross-modal pairs:

  • Skin I & II: Transferring from Dermoscopy (high-end) to Clinical photos (low-end) using the MRA-MIDAS and Derm7pt datasets.
  • Chest: Transferring from CT (CC-CCII) to Chest X-ray.
  • Breast: Transferring from Mammography (DMID) to Ultrasound (BUSI).

In all cases, the model was trained using a 16-shot setting on the high-end modality and tested zero-shot on the low-end modality.

Key Insights and Ideas

  • Structure Over Statistics: The most important "aha moment" is that we shouldn't just try to match features across modalities; we should match the relationships between classes. By preserving the manifold structure, the model retains the diagnostic logic even when the visual input changes drastically.
  • LLMs as Clinical Anchors: Using an LLM to generate a large ensemble of descriptions provides a robust "prior" that prevents the model from overfitting to the visual quirks of a specific dataset.
  • The "Harmonic Mean" Metric: The paper highlights the Harmonic Mean (H) of accuracy between modalities. This is a more honest metric for cross-modal transfer because it penalizes models that perform excellently on the source (high-end) but fail on the target (low-end), encouraging balanced generalization.

Key Results

K-MaT consistently outperformed state-of-the-art baselines like CoOp and BiomedCoOp:

  • Overall Performance: Achieved an average Harmonic Mean accuracy of 44.1% (up from BiomedCoOp’s 42.0%) and a macro-F1 of 36.2%.
  • Preventing Collapse: On the challenging breast imaging task, standard CoOp plummeted to 27.0% accuracy on the low-end (Ultrasound). K-MaT maintained a much more robust 38.4% on the low-end while keeping high-end accuracy at 73.1%.
  • Ablation Success: The ablation studies showed that the combination of MSC/CSC and the FGW alignment provided a 10.10% boost in Harmonic Mean accuracy over the baseline, proving that structural alignment is the key to cross-modal stability.

Limitations and Future Work

The authors acknowledge two main hurdles:

  1. Absolute Performance: While K-MaT is better than existing methods, the absolute accuracy on low-end modalities is still relatively low compared to supervised models trained directly on that data.
  2. Visual-Textual Gaps: In cases where the visual discrepancy between modalities is extreme (e.g., very different physics), purely text-anchored alignment might not be enough to bridge the gap.

Future Work: The team plans to explore incorporating more reliable visual signals (perhaps through self-supervised pre-training) to further stabilize the low-end transfer capabilities.

Summary updated

Abstract

Large-scale biomedical vision-language models (VLMs) adapted on high-end imaging (e.g., CT) often fail to transfer to frontline low-end modalities (e.g., radiography), collapsing into modality-specific shortcuts. We propose K-MaT (Knowledge-Anchored Manifold Transport), a prompt-learning framework that transfers decision structures to low-end modalities without requiring low-end training images. K-MaT factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport. We evaluate K-MaT on four cross-modal benchmarks, including dermoscopy, mammography to ultrasound, and CT to chest X-ray. K-MaT achieves state-of-the-art results, improving the average harmonic mean of accuracy to 44.1% (from BiomedCoOp's 42.0%) and macro-F1 to 36.2%. Notably, on the challenging breast imaging task, it mitigates the catastrophic forgetting seen in standard methods like CoOp (which drops to 27.0% accuracy on the low-end), preserving robust performance across modalities. Aligning prompt manifolds via optimal transport provides a highly effective route for the zero-shot cross-modal deployment of medical VLMs.
BibTeX
@misc{zeng2026k,
  author = {Jiajun Zeng and Shadi Albarqouni},
  title = {{K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging}},
  year = {2026},
  eprint = {2603.06340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.06340},
}
Figure