TL;DR
PACE-MoE is a new framework for medical visual grounding that coordinates what a model learns with where it stores that knowledge. By combining a policy-guided domain sampler (PACE) with a bank of domain-specific LoRA adapters (MoE), the system achieves a 3.64% relative improvement in localization accuracy across seven diverse medical imaging modalities.
What Problem Does This Paper Address?
Medical visual grounding—the task of linking clinical text descriptions to specific regions in a medical image—is notoriously difficult because of the sheer variety of imaging "domains" (e.g., CT, MRI, X-ray, Ultrasound).
Current post-training strategies for Multimodal Large Language Models (MLLMs) typically use a fixed mixture of data and a single set of shared parameters (like a standard LoRA adapter). The authors identify two major flaws in this "one-size-fits-all" approach:
- Static Sampling: Not all data is equally useful at all times. Early in training, a model might get zero reward from a complex domain, providing no useful feedback. Later, that same domain might be the only thing left to learn.
- Parameter Interference: Forcing a single adapter to learn the distinct visual features of seven different modalities can lead to sub-optimal performance as updates from one domain (like Ultrasound) might conflict with another (like Dermoscopy).
Methods
The authors propose PACE-MoE, which introduces "data-parameter routing" to the Group Relative Policy Optimization (GRPO) pipeline.
1. PACE: Policy-guided Adaptive Curriculum
Instead of a fixed data mix, PACE uses a small "probe set" to evaluate the model's current performance every few updates. It calculates three key statistics per domain:
- Mean Reward: How well is the model doing?
- Reward Dispersion: Is there enough variety in the responses to provide a learning signal?
- Validity: Is the model following the required output format?
These stats guide a three-stage curriculum:
- Admission: Start with domains the model can actually learn from, then progressively add others.
- Replay: Keep earlier domains in the mix to prevent forgetting.
- All-domain Balancing: A final stage that gives extra attention to domains where the model is still struggling.
2. Domain-Specific LoRA (MoE)
The framework replaces the single shared LoRA adapter with a bank of seven lightweight, domain-specific adapters (one per modality).
As shown in Figure 2, the domain label of a query (e.g., "CT") acts as a hard router. It selects the corresponding adapter for both generating the response and applying the RL update. This ensures that domain-specific knowledge is isolated, reducing interference while keeping the massive backbone model frozen and shared.
Datasets
The study utilizes the U-MRG dataset category-shift protocol, covering seven imaging domains: CT, MRI, X-ray, Ultrasound, Endoscopy, Dermoscopy, and Mammography.
- Training: U-MRG-6K (6,590 gradient-bearing examples).
- Evaluation: A held-out test set of 1,297 unique samples.
- Probing: A tiny, fixed buffer of 56 examples (8 per modality) used solely for domain scoring.
Key Insights and Ideas
- Difficulty is Dynamic: A domain isn't just "hard" or "easy"; its training value depends on the model's current state. PACE treats domain selection as a non-stationary problem.
- Coordinated Routing: The core "aha moment" is that data allocation and parameter adaptation should be linked. If you decide to train more on X-rays, those updates should go to the X-ray specialist adapter.
- Verification as a Signal: Because medical grounding has verifiable outputs (bounding boxes), the authors can use GRPO to optimize the model based on spatial overlap (IoU) without needing expensive human-in-the-loop feedback during the RL phase.
Key Results
PACE-MoE consistently outperformed the vanilla GRPO baseline across multiple metrics:
- Localization Accuracy: Achieved a 36.12% mean box IoU, representing a 3.64% relative improvement over the Vanilla baseline.
- Sampling Benefit: Even without the domain-specific adapters, the PACE adaptive sampler alone improved IoU by 2.07% compared to fixed sampling.
- Qualitative Improvements: As seen in Figure 3, PACE-MoE often corrects "anatomical hallucinations"—for example, correctly identifying the right kidney in a CT scan where the baseline model failed entirely.
The model showed particularly strong gains in smaller, specialized domains like Mammography (+10.21% relative gain) and Ultrasound (+4.63%).
Limitations and Future Work
- Label Dependency: The system relies on accurate imaging-domain labels. The authors note that performance on noisy, mixed, or unlabeled data remains an open question.
- Computational Overhead: While the adapters are lightweight, the "probing" phase (generating responses to evaluate domain value) adds to the total training time.
- Scope: The current study focuses on box and point grounding. Future work could extend this to full mask segmentation quality and direct clinical utility assessments.
Summary updated