Adapting General Vision Foundation Models to Medical Imaging

129 concepts111 papers4 proposals

Vision foundation models like SAM, DINOv2, and CLIP have achieved remarkable zero-shot capabilities by learning from billions of natural images. However, when deployed in clinical environments, these models encounter a steep performance drop. A model trained to segment cars and dogs frequently stumbles when faced with a glioblastoma on an MRI or a polyp in an endoscopic video.

This drop is caused by a profound natural-to-medical domain gap. Natural images are characterized by high contrast, rich color (RGB) channels, and explicit object boundaries. Medical images, by contrast, are often grayscale, exhibit low contrast, contain high noise or specular reflections, and require an understanding of complex 3D volumes or microscopic structures.

To bridge this gap without losing the rich structural priors embedded within foundation models, researchers have developed specialized adaptation strategies. Rather than training massive models from scratch or performing full fine-tuning—which is computationally prohibitive and risks catastrophic forgetting—the field has converged on parameter-efficient realignment, dimensional expansion, and automated clinical prompting.


The Failure of Frozen Features

When transferring a foundation model to a new domain, a common baseline is linear probing: freezing the backbone and training a small classification or segmentation head on top. However, as Joana Palés Huix et al. demonstrate in Are Natural Domain Foundation Models Useful for Medical Image Classification?, freezing the weights of general-purpose models during cross-domain transfer incurs a significant performance penalty.

The reason lies in feature misalignment. The intermediate layers of a frozen natural-image model look for semantic concepts like "fur," "wheels," or "sky." In a histopathology slide or a CT scan, the critical diagnostic features are subtle variations in cellular morphology, tissue density, or grayscale intensity gradients. As noted by Jun Ma et al. in Segment anything in medical images, this lack of domain-specific intuition causes general models to produce biologically implausible shapes or miss low-contrast tissue boundaries entirely.


Parameter-Efficient Feature Realignment

To alter the model's internal representations without destroying its pre-trained geometric priors, researchers leverage Parameter-Efficient Fine-Tuning (PEFT), primarily through lightweight adapter modules.

Instead of updating all billions of parameters, adapters inject small, trainable bottleneck layers into the frozen backbone. These layers act as localized coordinate transformers: they accept the generalized features from the foundation model and project them into a subspace optimized for clinical textures.

[Input Image] ---> [ Frozen Foundation Layer ] ---> [ Trainable Adapter ] ---> [ Next Layer ]
                          |                                 ^
                          +--------(Low-Rank Steering)------+

For example, Beilei Cui et al. utilize Low-Rank Adaptation (LoRA) in Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery. By training a mere 0.17% of the model's total parameters, they successfully steer DINOv2 to handle the unique lighting, fluid reflections, and lack of rigid textures characteristic of surgical environments.

To maximize the efficiency of these adapters, Zelin Peng et al. introduce a cross-block orchestration framework in Parameter Efficient Fine-Tuning via Cross Block Orchestration for Segment Anything Model. Rather than treating each layer's adaptation in isolation, their method allows different blocks of the Vision Transformer (ViT) to communicate during tuning. This global coordination breaks the limitations of layer-by-layer adaptation, enabling precise feature realignment across highly divergent domains like pathology and radiology with minimal parameter overhead.


Dimensional Expansion: Moving from 2D to 3D

A structural mismatch between natural and medical imaging is dimensionality. Prominent foundation models are inherently 2D, designed for flat images. Clinical diagnostics, however, rely heavily on 3D spatial context, such as the contiguous slices of a CT or MRI volume. Processing a 3D volume slice-by-slice using a purely 2D model discards inter-slice continuity, resulting in fragmented, inconsistent segmentations along the Z-axis.

To resolve this, researchers inject volumetric awareness directly into 2D architectures. In MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation, Cheng Chen et al. integrate lightweight 3D adapters into the 2D transformer blocks of the Segment Anything Model (SAM). These adapters extract features across adjacent slices, allowing the model to track anatomical structures volumetrically while preserving the powerful 2D feature extraction capabilities learned during large-scale pre-training.

Slice t-1  [ 2D ViT Block ] ---\
                                \
Slice t    [ 2D ViT Block ] -----[ 3D Adapter Fuses Spatial Context ] ---> Volumetric Feature
                                /
Slice t+1  [ 2D ViT Block ] ---/

Alternatively, this dimensional gap can be bridged at the data-propagation level. In the single-author work Slide-SAM: Medical SAM Meets Sliding Window, Heqin Zhu introduces a sliding-window mechanism that propagates segmentation masks across sequential 3D slices. By prompting only a single central slice, the framework leverages the model's sequential context to automatically map out the entire 3D volume, drastically reducing manual annotation overhead.


Automating Clinical Intent via Prompt Engineering

Interactive foundation models like SAM rely on user-provided prompts (such as bounding boxes or point clicks) to isolate objects. In a high-throughput clinical workflow, requiring a radiologist to manually click on every target structure is impractical. Furthermore, zero-shot prompting often fails because general models lack an understanding of clinical intent; they do not know whether a click signifies an entire organ, a specific lesion, or a sub-region of tissue.

To automate this process, researchers encode clinical intent directly into the prompt space. Xian Lin et al. address this in SAMCT: Segment Any CT Allowing Labor-Free Task-Indicator Prompts by introducing a "task-indicator" prompt encoder. Instead of relying on manual human clicks, the model uses an automated parallel branch to generate semantic embeddings that specify the target anatomical structure, effectively telling the frozen SAM backbone what to look for based on the clinical task.

Taking automation a step further, Xiuqi Zheng et al. propose a progressive refinement strategy in Curriculum Prompting Foundation Models for Medical Image Segmentation. Their framework mimics the diagnostic strategy of a clinician by implementing a curriculum that transitions from coarse bounding boxes to fine-grained edge points. By using intermediate segmentation masks as a bridge, the system iteratively refines its own prompts, eliminating human intervention while guiding the foundation model to resolve highly ambiguous tissue boundaries.

Key Foundational Work

A Benchmark of Medical Out of Distribution Detection (2020) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_domain_gap_in_foundation_models_12bb8a49,
  title = {domain gap in foundation models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-gap-in-foundation-models/rdr_12bb8a49fa354ec0b7dfa450c73f0a63},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_domain_gap_in_foundation_models_12bb8a49_v1,
  title = {domain gap in foundation models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-gap-in-foundation-models/rdr_12bb8a49fa354ec0b7dfa450c73f0a63/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake