vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs

Minye Shao, Sihan Guo, Xinrun Li, Xingyu Miao, Haoran Duan, Yang Long

TL;DR

vMFCoOp is a novel framework that bridges the semantic gap between Large Language Models (LLMs) and biomedical Vision-Language Models (VLMs) by aligning them on a unified hyperspherical manifold. By modeling embeddings as directional distributions (von Mises-Fisher), it achieves state-of-the-art few-shot classification and superior interpretability across 14 diverse medical datasets.

What Problem Does This Paper Address?

Adapting general-purpose Vision-Language Models (like CLIP) to the biomedical domain is notoriously difficult. Medical images are visually complex, highly structured, and require deep domain expertise. While "Prompt Learning" (e.g., CoOp) has emerged as a way to adapt these models without full fine-tuning, two major hurdles remain:

  1. The Semantic Gap: There is a fundamental misalignment between the "logic" of an LLM (used to generate medical descriptions) and the embedding space of a VLM backbone. They are trained on different data with different architectures.
  2. Euclidean Limitations: Most methods optimize in flat Euclidean space. However, high-dimensional embeddings in models like CLIP are typically normalized, meaning their semantic relationships are defined by angles rather than straight-line distances. Standard optimization often fails to capture this "directional" geometry, leading to unstable performance in few-shot scenarios.

Methods

The authors introduce vMFCoOp, which moves prompt optimization from flat space to a Unified Hyperspherical Manifold ($S^{d-1}$).

Key Innovations:

  • vMF Inverse Mapping: The framework treats embeddings as samples from a von Mises-Fisher (vMF) distribution—essentially the "Gaussian distribution" of a sphere. It estimates a "CLIP Semantic Anchor Field" (from the model's vocabulary) and an "LLM Semantic Prototype Field" (from LLM-generated descriptions).
  • Unified Semantic Anchors: These two fields are fused into a single target "anchor" for each medical class. This acts as a North Star, guiding the learnable prompts toward a point where LLM knowledge and CLIP's visual features reach equilibrium.
  • Three Hierarchical Constraints:
    1. Semantic Anchor Loss ($\mathcal{L}_{anc}$): Constrains learnable prompts to stay near the unified anchors while allowing for dynamic "navigation" on the sphere.
  • Spherical Contrastive Loss ($\mathcal{L}_{sc}$): Uses an annealing temperature schedule to progressively sharpen decision boundaries, pushing different classes further apart angularly.
  • Symmetric Cross-Entropy Loss ($\mathcal{L}_{SCE}$): Jointly optimizes the probability of correct image-prompt pairings to ensure the model is both certain and discriminative.

Datasets

The authors conducted an extensive evaluation across 14 biomedical datasets, covering 12 modalities and 13 anatomical regions:

  • UK Biobank: Cardiac Cine MRI, Liver MRI, and Pancreas MRI (using ICD-10 codes for fine-grained labels).
  • Public Benchmarks: Kvasir (Endoscopy), CTKidney, BTMRI (Brain MRI), LC25000 & CHMNIST (Histopathology), DermaMNIST (Dermoscopy), RETINA (Fundus), OCTMNIST (OCT), BUSI (Ultrasound), COVID-QU-Ex (X-ray), and KneeXray.

Key Insights and Ideas

  • Geometry Matters: The core "aha moment" is that in high-dimensional normalized spaces, direction is everything. By forcing the model to respect the hyperspherical geometry, the authors mitigate the "modality gap" that usually plagues multimodal models.
  • Equilibrium via Anchoring: Instead of letting the prompt drift freely during training, anchoring it to a fusion of LLM and CLIP priors prevents the model from "hallucinating" or overfitting to the very small number of samples available in few-shot learning.
  • Overcoming Inductive Bias: The method effectively balances the LLM's "textual bias" with CLIP's "visual bias," leading to a more robust representation that generalizes better to unseen (novel) classes.

Key Results

  • Superior Accuracy: vMFCoOp consistently outperformed all baselines (including CoOp and BiomedCoOp). It showed a 7.29% relative gain at 4-shot and a 5.22% gain at 64-shot.
  • Robust Generalization: In "Base-to-Novel" tests (training on some diseases and testing on others), it achieved a Harmonic Mean of 77.35%, surpassing the previous state-of-the-art by roughly 4.8%.
  • Clinical Interpretability: Using saliency maps (gScoreCAM), the authors showed that vMFCoOp accurately localizes lesions (like tiny kidney stones or rare mediastinal tumors) that other models completely miss or misidentify due to semantic bias.
  • Model Agnostic: The framework proved effective regardless of which LLM (GPT-4, Claude, Qwen, DeepSeek) or CLIP backbone (BiomedCLIP, MedCLIP, etc.) was used.

Limitations and Future Work

  • Task Scope: The current work focuses primarily on classification. The authors plan to extend the framework to more complex tasks like medical image segmentation and visual grounding.
  • Domain Expansion: While highly successful in the biomedical field, the authors suggest that the hyperspherical alignment strategy could be applied to natural image domains to improve general VLM robustness.
  • Theoretical Depth: Future iterations aim to provide even deeper theoretical insights into the convergence properties of optimization on non-Euclidean manifolds.

Summary updated

Abstract

Recent advances in context optimization (CoOp) guided by large language model (LLM)-distilled medical semantic priors offer a scalable alternative to manual prompt engineering and full fine-tuning for adapting biomedical CLIP-based vision-language models (VLMs). However, prompt learning in this context is challenged by semantic misalignment between LLMs and CLIP variants due to divergent training corpora and model architectures; it further lacks scalability across continuously evolving families of foundation models. More critically, pairwise multimodal alignment via conventional Euclidean-space optimization lacks the capacity to model unified representations or apply localized geometric constraints, which tends to amplify modality gaps in complex biomedical imaging and destabilize few-shot adaptation. In this work, we propose vMFCoOp, a framework that inversely estimates von Mises-Fisher (vMF) distributions on a shared Hyperspherical Manifold, aligning semantic biases between arbitrary LLMs and CLIP backbones via Unified Semantic Anchors to achieve robust biomedical prompting and superior few-shot classification. Grounded in three complementary constraints, vMFCoOp demonstrates consistent improvements across 14 medical datasets, 12 medical imaging modalities, and 13 anatomical regions, outperforming state-of-the-art methods in accuracy, generalization, and clinical applicability. This work aims to continuously expand to encompass more downstream applications, and the corresponding resources are intended to be shared through https://github.com/VinyehShaw/UniEqui.
BibTeX
@inproceedings{shao2026vmfcoop,
  author = {Minye Shao and Sihan Guo and Xinrun Li and Xingyu Miao and Haoran Duan and Yang Long},
  title = {{vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i11.37839},
}
Figure