TL;DR
The paper introduces MoR (Mixture-of-Rewards), a federated learning framework designed to align heterogeneous Vision-Language Models (VLMs) without sharing private data or model parameters. By replacing traditional parameter aggregation with a learned, dynamic routing mechanism over client-specific reward models, MoR successfully navigates conflicting client preferences and diverse model architectures while maintaining strict data privacy.
What Problem Does This Paper Address?
Aligning Vision-Language Models (VLMs) with human preferences (e.g., safety, helpfulness) typically relies on centralized reinforcement learning from human feedback (RLHF). However, in privacy-sensitive domains like healthcare or finance, centralizing data is often impossible due to strict regulations.
While Federated Learning (FL) keeps data local, standard FL approaches break down under the extreme heterogeneity of real-world deployments. Clients often use completely different model architectures due to varying hardware constraints. More fundamentally, they often have conflicting preference criteria. As illustrated below, for the exact same driving scene, a detail-focused client might prefer a highly descriptive response, whereas a safety-focused client demands a concise hazard warning.
Directly averaging parameters or forcing a single global reward model under these conditions obscures client specialization and degrades performance.
Methods
To coordinate these diverse supervision signals, the authors propose MoR, which operates in three decentralized stages:
- Reward Model Initialization: Each client independently trains a local Reward Model (RM) on its private preference data. To ensure scores are comparable across different clients, a local normalization step standardizes the reward outputs.
- Routing Network Federated Learning: Instead of aggregating full model parameters, the server coordinates the training of a lightweight global routing network ($g_\phi$) via standard federated averaging. This router learns to map multimodal input-response pairs to the most capable client RM.
- GRPO Policy Alignment with Online Router Updating: The central server optimizes a base VLM using Group Relative Policy Optimization (GRPO). To minimize communication overhead, MoR uses sparse routing—the router selects only the single most suitable client RM to score a given response group.
Crucially, because the VLM policy continuously evolves during GRPO training, an offline-trained router would eventually suffer from distribution mismatch. MoR solves this by framing online routing as a contextual bandit problem, updating the router online using Neural Thompson Sampling based on policy improvement feedback.
Datasets
The authors constructed a heterogeneous vision-language preference dataset by unifying and formatting several open-source benchmarks into pairwise preference data:
- POVID and VLFeedback: Used to build domain-specific clients focusing on Detailed Description, Medical Vision-Language Understanding, and OCR-like Visual Reasoning.
- MME-RealWorld: A high-resolution multiple-choice benchmark split by subtasks to introduce task-level heterogeneity across clients.
Key Insights and Ideas
- Preferences Over Parameters: When clients are structurally or behaviorally diverse, aggregating preferences via a mixture-of-experts routing layer provides a much cleaner and more effective collaboration interface than forcing parameter synchronization.
- The Bandit Fix for Distribution Shift: A static router fails when the underlying generative model changes. Treating the router as an online contextual bandit allows it to dynamically adapt to the VLM's shifting output distribution during reinforcement learning.
- Normalization Stabilizes Routing: Raw reward scores from independent models cannot be compared directly due to scale variance. Local normalization is a simple but non-negotiable prerequisite for stable global policy optimization.
Key Results
- In-Distribution Superiority: On the MME-RealWorld benchmark, MoR achieved an in-distribution average accuracy of 79.99%, outperforming standard
FedAvgby 10.29 percentage points and beating other decentralized baselines by up to 26.65%. - Cross-Category Generalization: When tested on eight unseen, held-out categories, MoR achieved the highest overall accuracy (70.07%), proving that the router learns transferable multimodal context associations rather than just memorizing client identities.
- Robust to Architecture Mixing: MoR successfully aligned clients running completely different VLM backbones (Qwen2-VL-2B, Qwen2.5-VL-3B, and LLaVA-OneVision-0.5B), maintaining stable performance where single-model baselines faltered.
- Privacy and Security: Under query-only membership inference and gradient inversion attacks, MoR demonstrated near-random attacker success, confirming excellent privacy preservation.
Limitations and Future Work
The authors note that while MoR drastically reduces parameter communication, it still requires exchanging multimodal query payloads (images and text) between the server and clients during the online GRPO evaluation phase. In bandwidth-constrained environments, transmitting high-resolution images represents a significant bottleneck. The authors point to visual payload optimization—such as feature caching, image compression, or transmitting compact token representations—as the primary direction for future work.
Summary updated