Convergent Performance, Divergent Mechanisms in LoRA-Adapted Vision Foundation Models for Brain MRI

Cecilia Diana-Albelda, Fiona Kekwick, Raghav Mehta, Aleksandra Kulbaka, Paul M. Matthews, Jesus Bescos, Álvaro García-Martín, Wenjia Bai, Bernhard KainzarXiv 2026 · arXiv

TL;DR

Adapting Vision Foundation Models (VFMs) to medical tasks using LoRA often yields similar performance across different backbones, but this "convergent performance" hides radically different internal adaptation mechanisms. This paper reveals that the most heavily modified layers are often not the most causally important, and that standard adaptation can inadvertently degrade a model's performance on unrelated clinical tasks—a risk that can be mitigated by restricting adaptation to early-to-mid layers.

What Problem Does This Paper Address?

In the era of foundation models, researchers often adapt pre-trained backbones (like DINO or CLIP) to specific medical tasks using Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA. Currently, model selection is almost entirely driven by downstream metrics: if two backbones achieve the same Dice score for brain tumor segmentation, they are treated as interchangeable.

However, this approach ignores the "black box" of how these models actually change internally. The authors ask: Does equal performance imply equal representational shift? More importantly, does adapting a model for one task (e.g., segmentation) break its ability to perform another (e.g., disease classification)? Understanding these internal dynamics is crucial for building robust, multi-purpose medical AI systems.

Methods

The researchers analyzed four ViT-B/16 backbones with different pre-training objectives and data domains:

  • Self-distillation: DINOv3 (natural images) and MerMED-FM (medical images).
  • Contrastive vision-language: SigLIP2 (natural) and BiomedCLIP (medical).

They adapted these models for brain tumor segmentation using LoRA and employed a suite of mechanistic interpretability tools to look under the hood:

  • Attention JS Divergence: Measuring how much the attention patterns shift per layer.
  • LoRA Weight-Change Ratio: Quantifying the magnitude of parameter updates.
  • Off-brain Attention Fraction: Tracking how much attention is redirected from background regions to the anatomy.
  • Activation Patching: A causal intervention where adapted activations from specific layers are injected into a frozen model to see which layers actually drive the performance gains.
Figure 1: Overview of the adaptation and analysis pipeline. (Panel A) Four brain MRI modalities (T1n, T1c, T2f, T2w) are fused via a trainable 1 × 1 convolution into a 3-channel input, processed by a frozen ViT-B/16 encoder with trainable LoRA modules applied to the query (Q) and value (V) projections of every transformer block in the default all-layer configuration. Components marked with \ (fusion convolution, LoRA modules, and segmentation head) are jointly trained on BraTS-GLI. (Panel B) Segmentation is evaluated on three BraTS datasets. For classification, the full encoder (including LoRA weights) remains frozen and only a logistic regression classifier (\) is fitted on each classification dataset using mean-pooled features from 𝓁12; the fusion convolution is bypassed (T1n input only). Four mechanistic interpretability tools are applied per layer to analyse the adaptation process: JS divergence, weight-change ratio, off-brain attention fraction, and activation patching.

Datasets

The models were trained and evaluated across six diverse datasets:

  • Segmentation: BraTS-GLI (adult glioma), BraTS-Africa (out-of-distribution generalizability), and BraTS-PED (pediatric glioma).
  • Classification: OASIS-1 and ADNI (binary and multi-class Alzheimer’s disease classification) used to test cross-task transfer via linear probing.

Key Insights and Ideas

  • Performance Convergence Masks Mechanistic Divergence: While all models reached a similar segmentation accuracy (78–81% Dice), their internal shifts were vastly different. For instance, SigLIP2 required massive attention shifts in late layers, while DINOv3 remained remarkably stable.
  • The Dissociation of Magnitude and Importance: A major "aha moment" was finding that the layers modified the most (typically the late layers) were not the ones causally responsible for the performance boost. Activation patching showed that early-to-mid layers (1–6) were sufficient to recover nearly all segmentation performance.
  • The "Medical Domain" Paradox: Surprisingly, models pre-trained on medical data (BiomedCLIP, MerMED-FM) did not necessarily adapt more efficiently. Even though they already "knew" to look at the brain, they required larger parameter updates in later layers to solve the specific segmentation task compared to the natural-domain DINOv3.
  • Cross-Task Fragility: Adapting for segmentation often "overwrites" features useful for classification. MerMED-FM, SigLIP2, and BiomedCLIP all saw performance drops in Alzheimer's classification after being tuned for tumor segmentation. DINOv3 was the lone exception, showing improved performance on both tasks.
Figure 3: Per-layer mechanistic analysis of LoRA adaptation. (a) Attention Jensen-Shannon divergence between frozen and adapted attention distributions. (b) LoRA weight-change ratio (‖Δ𝑊‖𝐹∕‖𝑊‖𝐹), shown separately for query (Q, solid) and value (V, dashed) projections on a log scale. (c) Frozen off-brain attention fraction, measuring the fraction of attention mass directed to off-brain patches before adaptation. All metrics are computed on the BraTS-GLI test set and averaged across patients and MRI slices. Shaded bands show ±1 standard deviation across 3 seeds (absent in (c), which uses only pre-trained weights).

Key Results

  • Segmentation: All models achieved high performance, with DINOv3 slightly leading. LoRA adaptation refined region boundaries significantly across the board.
  • Classification Degradation: MerMED-FM suffered the most, with its OASIS-1 classification AUC dropping from 0.878 (frozen) to 0.830 (adapted).
  • Layer-Selective Success: By restricting LoRA to only the first six layers (Early-layer LoRA), the researchers matched or approached full-layer segmentation performance while significantly reducing the degradation in classification tasks for the affected backbones.
Figure 2: Qualitative segmentation results for a representative axial slice from each dataset. Tumour sub-regions (enhancing, necrotic core, peritumoural edema) are overlaid on the T2f input. Each row corresponds to one dataset (BraTS-GLI, BraTS-Africa, BraTS-PED). For each backbone, the frozen (𝐹) and LoRA-adapted (𝐴) predictions are shown side by side. LoRA adaptation visibly refines region boundaries across all four backbones.

Limitations and Future Work

The study is currently limited to the ViT-B/16 architecture and the brain MRI domain. The authors note that it remains to be seen if these patterns hold for larger models (ViT-L/H), other imaging modalities (like CT or pathology), or different PEFT methods beyond LoRA.

Future work will likely explore whether these mechanistic insights can lead to automated, "smart" adapter placement strategies that optimize for both task performance and representational robustness without requiring manual layer selection. Not discussed in the paper.

Summary updated

Abstract

Adapting Vision Foundation Models (VFMs) to medical imaging via LoRA has become standard practice, yet model selection is guided almost entirely by downstream task metrics, treating backbones with similar performance as interchangeable. We investigate whether equal performance implies equal internal representational shift. We analyse four ViT-B/16 backbones, DINOv3, SigLIP2, MerMED-FM, and BiomedCLIP, pre-trained with different objectives (self-distillation vs. contrastive vision-language) and on different data domains (natural vs. medical images). We adapt each to brain tumour segmentation and evaluate on six datasets covering both segmentation and Alzheimer's disease classification. Using attention divergence, weight-change analysis, and activation patching, we find that the four models converge to similar segmentation performance (mean Dice 78-81%) through markedly different internal dynamics, yet diverge in cross-task transfer: segmentation adaptation improves Alzheimer's disease classification for DINOv3 but degrades it for the other three backbones. We reveal a dissociation between modification magnitude and causal importance: the most heavily modified layers are not the causally relevant ones, with early-to-mid layers identified as sufficient for segmentation recovery. Restricting LoRA to these layers matches or approaches full-layer segmentation while reducing classification degradation in the three affected backbones, providing a mechanistically grounded strategy that is more parameter-efficient and more robust than standard full-layer adaptation.
BibTeX
@misc{dianaalbelda2026convergent,
  author = {Cecilia Diana-Albelda and Fiona Kekwick and Raghav Mehta and Aleksandra Kulbaka and Paul M. Matthews and Jesus Bescos and Álvaro García-Martín and Wenjia Bai and Bernhard Kainz},
  title = {{Convergent Performance, Divergent Mechanisms in LoRA-Adapted Vision Foundation Models for Brain MRI}},
  year = {2026},
  eprint = {2610.28255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.28255},
}
Figure