vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs

Minye Shao, Sihan Guo, Xinrun Li, Xingyu Miao, Haoran Duan, Yang Long

Introduction

Adapting Vision-Language Models (VLMs) like CLIP to the specialized domain of medical imaging presents a unique set of challenges. While prompt learning techniques such as CoOp have shown promise in general domains, they often falter when faced with the fine-grained, highly structured semantics of biomedical data. A primary hurdle is the semantic misalignment between Large Language Models (LLMs) used for prompt generation and the CLIP vision-language space. These models, trained on disparate datasets with different architectures, do not naturally share a representational language.

Furthermore, standard prompt optimization typically occurs in flat Euclidean space. However, because CLIP embeddings are normalized, they inherently reside on the surface of a hypersphere. Optimizing in Euclidean space ignores this native geometry, leading to instability and overfitting, particularly in few-shot scenarios where labeled data is scarce. To address these issues, we introduce vMFCoOp, a framework designed to align heterogeneous models on a Unified Hyperspherical Manifold using von Mises-Fisher (vMF) distributions.

The vMFCoOp Framework

The core innovation of vMFCoOp lies in its treatment of the optimization space as a non-Euclidean manifold. By utilizing the vMF distribution—the spherical analogue of the Gaussian distribution—the framework can effectively model directional data.

The vMFCoOp architecture aligns LLM and CLIP representations on a unified hyperspherical manifold using vMF distributions and specialized losses.

The framework operates in two primary phases:

  1. Unified Semantic Anchors: vMFCoOp extracts vocabulary embeddings from CLIP and prompt embeddings from an LLM. It then fits vMF distributions to both sets of embeddings. By calculating the mean direction and concentration, the framework fuses these into "Unified Semantic Anchors" for each medical class. This process reconciles the semantic biases of both models, creating a stable reference point for learning.
  2. Hyperspherical Optimization: The learnable prompts are optimized on the hypersphere through three specific constraints:
  3. Semantic Anchor Loss ($\mathcal{L}_{anc}$): This steers prompt embeddings toward their unified anchors, ensuring they remain semantically grounded.
  4. Spherical Contrastive Loss ($\mathcal{L}_{sc}$): This increases the angular margin between categories by pushing prompts away from irrelevant "distractor" classes.
  5. Symmetric Cross-Entropy Loss ($\mathcal{L}_{SCE}$): This calibrates prediction certainty by tightly coupling visual and prompt embeddings.

Experimental Evaluation

vMFCoOp was rigorously tested across 14 biomedical imaging datasets, covering 12 modalities (e.g., MRI, CT, X-ray, Ultrasound, Histopathology) and 13 anatomical regions. This extensive evaluation ensures the framework's robustness across diverse clinical scenarios, from identifying kidney stones in CT scans to detecting lesions in brain MRIs.

Quantitative Performance

The results demonstrate that vMFCoOp consistently outperforms state-of-the-art (SOTA) baselines, including CoOp, KgCoOp, and BiomedCoOp. Notably, it achieved a 7.29% relative accuracy gain at 4-shot and a 5.22% gain at 64-shot over the nearest competitor. Unlike standard methods that often plateau as more few-shot examples are added, vMFCoOp's geometric constraints allow for sustained improvement with additional data.

In terms of Base-to-Novel Generalization, vMFCoOp showed exceptional performance. When trained on a set of base classes and evaluated on entirely unseen novel classes, it achieved an average Harmonic Mean accuracy of 77.35%, marking a significant ~4.8% absolute improvement over previous SOTA methods.

Visual Interpretability

Beyond classification accuracy, vMFCoOp provides superior visual interpretability. Using gScoreCAM to generate saliency maps, we compared the localization capabilities of various methods.

Saliency maps across different modalities show that vMFCoOp accurately localizes fine-grained lesions compared to other prompt learning methods.

As shown in the figure above, vMFCoOp accurately pinpoints fine-grained lesions, such as tiny kidney stones or specific tumors. In contrast, competing methods often suffer from modality bias, highlighting broad, irrelevant anatomical structures rather than the specific pathology. This precision is critical for clinical trust and utility.

Conclusion

By respecting the native hyperspherical geometry of VLMs and statistically bridging the gap between LLMs and CLIP, vMFCoOp sets a new standard for medical prompt learning. It addresses the fundamental issues of semantic misalignment and geometric instability, resulting in a model that is both more accurate and more interpretable. Future work will explore the scalability of this hyperspherical approach to broader multimodal foundation models and a wider array of downstream tasks.

Summary updated

BibTeX
@inproceedings{shao2026vmfcoop,
  author = {Minye Shao and Sihan Guo and Xinrun Li and Xingyu Miao and Haoran Duan and Yang Long},
  title = {{vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i11.37839},
}
Figure