PACE-MoE: Adaptive Domain Sampling and Domain-Specific LoRA for Medical Visual Grounding

Zhenxuan Wang, Jisheng Dang, Ronghao Lin, Bimei Wang, Jingwen Zhao, Wenjun Ma, Bin Hu, Qi Tian, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

PACE-MoE is a new framework for medical visual grounding that coordinates what a model learns with where it stores that knowledge. By combining a policy-guided domain sampler (PACE) with a bank of domain-specific LoRA adapters (MoE), the system achieves a 3.64% relative improvement in localization accuracy across seven diverse medical imaging modalities.

What Problem Does This Paper Address?

Medical visual grounding—the task of linking clinical text descriptions to specific regions in a medical image—is notoriously difficult because of the sheer variety of imaging "domains" (e.g., CT, MRI, X-ray, Ultrasound).

Current post-training strategies for Multimodal Large Language Models (MLLMs) typically use a fixed mixture of data and a single set of shared parameters (like a standard LoRA adapter). The authors identify two major flaws in this "one-size-fits-all" approach:

  1. Static Sampling: Not all data is equally useful at all times. Early in training, a model might get zero reward from a complex domain, providing no useful feedback. Later, that same domain might be the only thing left to learn.
  2. Parameter Interference: Forcing a single adapter to learn the distinct visual features of seven different modalities can lead to sub-optimal performance as updates from one domain (like Ultrasound) might conflict with another (like Dermoscopy).

Methods

The authors propose PACE-MoE, which introduces "data-parameter routing" to the Group Relative Policy Optimization (GRPO) pipeline.

1. PACE: Policy-guided Adaptive Curriculum

Instead of a fixed data mix, PACE uses a small "probe set" to evaluate the model's current performance every few updates. It calculates three key statistics per domain:

  • Mean Reward: How well is the model doing?
  • Reward Dispersion: Is there enough variety in the responses to provide a learning signal?
  • Validity: Is the model following the required output format?

These stats guide a three-stage curriculum:

  • Admission: Start with domains the model can actually learn from, then progressively add others.
  • Replay: Keep earlier domains in the mix to prevent forgetting.
  • All-domain Balancing: A final stage that gives extra attention to domains where the model is still struggling.

2. Domain-Specific LoRA (MoE)

The framework replaces the single shared LoRA adapter with a bank of seven lightweight, domain-specific adapters (one per modality).

Fig. 2. Overview of PACE-MoE. (a) Architecture of PACE-MoE. Current-policy rollouts on a fixed seven-domain probe buffer estimate reward, within-query dispersion, and output validity. These statistics update the domain sampling distribution through domain admission, replay, and a final all-domain stage. The sampled label gt,b selects a query from Dg and activates the corresponding rank-16 FFN LoRA adapter ϕg. (b) Optimization pipeline. The selected adapter is updated using the shared spatial reward and GRPO objective, while the MedReasoner backbone and MedSAM2 remain frozen. At each probe event, the updated policy is evaluated again to determine the next domain sampling distribution.

As shown in Figure 2, the domain label of a query (e.g., "CT") acts as a hard router. It selects the corresponding adapter for both generating the response and applying the RL update. This ensures that domain-specific knowledge is isolated, reducing interference while keeping the massive backbone model frozen and shared.

Datasets

The study utilizes the U-MRG dataset category-shift protocol, covering seven imaging domains: CT, MRI, X-ray, Ultrasound, Endoscopy, Dermoscopy, and Mammography.

  • Training: U-MRG-6K (6,590 gradient-bearing examples).
  • Evaluation: A held-out test set of 1,297 unique samples.
  • Probing: A tiny, fixed buffer of 56 examples (8 per modality) used solely for domain scoring.

Key Insights and Ideas

  • Difficulty is Dynamic: A domain isn't just "hard" or "easy"; its training value depends on the model's current state. PACE treats domain selection as a non-stationary problem.
  • Coordinated Routing: The core "aha moment" is that data allocation and parameter adaptation should be linked. If you decide to train more on X-rays, those updates should go to the X-ray specialist adapter.
  • Verification as a Signal: Because medical grounding has verifiable outputs (bounding boxes), the authors can use GRPO to optimize the model based on spatial overlap (IoU) without needing expensive human-in-the-loop feedback during the RL phase.

Key Results

PACE-MoE consistently outperformed the vanilla GRPO baseline across multiple metrics:

  • Localization Accuracy: Achieved a 36.12% mean box IoU, representing a 3.64% relative improvement over the Vanilla baseline.
  • Sampling Benefit: Even without the domain-specific adapters, the PACE adaptive sampler alone improved IoU by 2.07% compared to fixed sampling.
  • Qualitative Improvements: As seen in Figure 3, PACE-MoE often corrects "anatomical hallucinations"—for example, correctly identifying the right kidney in a CT scan where the baseline model failed entirely.
Fig. 3. Qualitative comparison across seven imaging domains using the same test image and query for each pair. Top: Vanilla-25. Bottom: PACE-MoE. Solid green and dashed orange boxes denote ground truth and predictions, respectively. CT (a) and X-ray (c) retain the full image; the other columns use identical crops for both methods that retain all compared boxes. IoU is reported as a percentage and computed in the original image coordinates.

The model showed particularly strong gains in smaller, specialized domains like Mammography (+10.21% relative gain) and Ultrasound (+4.63%).

Limitations and Future Work

  • Label Dependency: The system relies on accurate imaging-domain labels. The authors note that performance on noisy, mixed, or unlabeled data remains an open question.
  • Computational Overhead: While the adapters are lightweight, the "probing" phase (generating responses to evaluate domain value) adds to the total training time.
  • Scope: The current study focuses on box and point grounding. Future work could extend this to full mask segmentation quality and direct clinical utility assessments.

Summary updated

Abstract

Medical visual grounding requires models to align clinical descriptions with spatial evidence across heterogeneous imaging domains. However, existing multi-domain post-training strategies typically rely on fixed domain sampling and shared parameter updates, overlooking the need to adapt both which domains receive training and where their updates are applied. To address this issue, we propose PACE-MoE, a policy-guided framework that jointly performs adaptive domain sampling and domain-specific adaptation for medical visual grounding. PACE dynamically assesses the training value of each imaging domain from current-policy feedback and progressively adjusts domain coverage through domain admission, replay, and final all-domain balancing. Meanwhile, domain-specific LoRA adapters route updates to the corresponding imaging domains while preserving the pretrained backbone and visual encoder. This coordinated design aligns training-data allocation with parameter adaptation without modifying the original reward formulation or Group Relative Policy Optimization (GRPO) objective. Experiments across seven medical imaging domains show that PACE-MoE consistently improves grounding performance over fixed-domain training, achieving a 3.64% relative improvement in mean box IoU over the Vanilla baseline. Under matched domain-specific adapter architectures, adaptive sampling alone further improves mean box IoU by 2.07%, demonstrating the benefit of policy-guided domain allocation. These results establish coordinated data-parameter routing as an effective strategy for multi-domain medical visual grounding.
BibTeX
@misc{wang2026pace,
  author = {Zhenxuan Wang and Jisheng Dang and Ronghao Lin and Bimei Wang and Jingwen Zhao and Wenjun Ma and Bin Hu and Qi Tian and Tat-Seng Chua},
  title = {{PACE-MoE: Adaptive Domain Sampling and Domain-Specific LoRA for Medical Visual Grounding}},
  year = {2026},
  eprint = {2610.27656},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.27656},
}
Figure