Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models

Shule Lu, Yujing Wang, Hainan Zhang, Xiaoshan Yang, Hongwei Zheng, Yongxin Tong, Changsheng Xu, Zhiming Zheng

TL;DR

The paper introduces MoR (Mixture-of-Rewards), a federated learning framework designed to align heterogeneous Vision-Language Models (VLMs) without sharing private data or model parameters. By replacing traditional parameter aggregation with a learned, dynamic routing mechanism over client-specific reward models, MoR successfully navigates conflicting client preferences and diverse model architectures while maintaining strict data privacy.

What Problem Does This Paper Address?

Aligning Vision-Language Models (VLMs) with human preferences (e.g., safety, helpfulness) typically relies on centralized reinforcement learning from human feedback (RLHF). However, in privacy-sensitive domains like healthcare or finance, centralizing data is often impossible due to strict regulations.

While Federated Learning (FL) keeps data local, standard FL approaches break down under the extreme heterogeneity of real-world deployments. Clients often use completely different model architectures due to varying hardware constraints. More fundamentally, they often have conflicting preference criteria. As illustrated below, for the exact same driving scene, a detail-focused client might prefer a highly descriptive response, whereas a safety-focused client demands a concise hazard warning.

Figure 1: Illustration of preference heterogeneity in feder- ated VLM alignment. For the same driving image and query, a detail-focused client prefers a comprehensive description, while a safety-focused client prefers a concise hazard-aware response, leading to conflicting reward signals for a single global reward model.

Directly averaging parameters or forcing a single global reward model under these conditions obscures client specialization and degrades performance.

Methods

To coordinate these diverse supervision signals, the authors propose MoR, which operates in three decentralized stages:

  1. Reward Model Initialization: Each client independently trains a local Reward Model (RM) on its private preference data. To ensure scores are comparable across different clients, a local normalization step standardizes the reward outputs.
  2. Routing Network Federated Learning: Instead of aggregating full model parameters, the server coordinates the training of a lightweight global routing network ($g_\phi$) via standard federated averaging. This router learns to map multimodal input-response pairs to the most capable client RM.
  3. GRPO Policy Alignment with Online Router Updating: The central server optimizes a base VLM using Group Relative Policy Optimization (GRPO). To minimize communication overhead, MoR uses sparse routing—the router selects only the single most suitable client RM to score a given response group.

Crucially, because the VLM policy continuously evolves during GRPO training, an offline-trained router would eventually suffer from distribution mismatch. MoR solves this by framing online routing as a contextual bandit problem, updating the router online using Neural Thompson Sampling based on policy improvement feedback.

Figure 2: Overview of our proposed federated framework architecture. The framework is composed of three interconnected stages: (A) Reward Model Initialization: Individual clients independently train local reward models (RMs) on their private, domain-specific datasets. (B) Routing Network Federated Learning: A global routing mechanism is collaboratively trained across clients to learn how to dispatch inputs to the most capable local RM. (C) GRPO Policy Alignment: The final multimodal policy is optimized using Group Relative Policy Optimization (GRPO), guided by the high-quality reward signals selected by the trained router.

Datasets

The authors constructed a heterogeneous vision-language preference dataset by unifying and formatting several open-source benchmarks into pairwise preference data:

  • POVID and VLFeedback: Used to build domain-specific clients focusing on Detailed Description, Medical Vision-Language Understanding, and OCR-like Visual Reasoning.
  • MME-RealWorld: A high-resolution multiple-choice benchmark split by subtasks to introduce task-level heterogeneity across clients.

Key Insights and Ideas

  • Preferences Over Parameters: When clients are structurally or behaviorally diverse, aggregating preferences via a mixture-of-experts routing layer provides a much cleaner and more effective collaboration interface than forcing parameter synchronization.
  • The Bandit Fix for Distribution Shift: A static router fails when the underlying generative model changes. Treating the router as an online contextual bandit allows it to dynamically adapt to the VLM's shifting output distribution during reinforcement learning.
  • Normalization Stabilizes Routing: Raw reward scores from independent models cannot be compared directly due to scale variance. Local normalization is a simple but non-negotiable prerequisite for stable global policy optimization.

Key Results

  • In-Distribution Superiority: On the MME-RealWorld benchmark, MoR achieved an in-distribution average accuracy of 79.99%, outperforming standard FedAvg by 10.29 percentage points and beating other decentralized baselines by up to 26.65%.
  • Cross-Category Generalization: When tested on eight unseen, held-out categories, MoR achieved the highest overall accuracy (70.07%), proving that the router learns transferable multimodal context associations rather than just memorizing client identities.
  • Robust to Architecture Mixing: MoR successfully aligned clients running completely different VLM backbones (Qwen2-VL-2B, Qwen2.5-VL-3B, and LLaVA-OneVision-0.5B), maintaining stable performance where single-model baselines faltered.
  • Privacy and Security: Under query-only membership inference and gradient inversion attacks, MoR demonstrated near-random attacker success, confirming excellent privacy preservation.

Limitations and Future Work

The authors note that while MoR drastically reduces parameter communication, it still requires exchanging multimodal query payloads (images and text) between the server and clients during the online GRPO evaluation phase. In bandwidth-constrained environments, transmitting high-resolution images represents a significant bottleneck. The authors point to visual payload optimization—such as feature caching, image compression, or transmitting compact token representations—as the primary direction for future work.

Summary updated

Abstract

Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance, yet strict data-sharing constraints render centralized training infeasible. Federated Learning mitigates this issue by enabling decentralized training, but practical deployments face challenges due to client heterogeneity in computational resources, application requirements, and model architectures. Under extreme model and data heterogeneity, replacing parameter aggregation with preference-based collaboration offers a more suitable interface, as it eliminates the need for direct parameter or data exchange. Motivated by this, we propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneous VLMs. In MoR, each client locally trains a reward model from local preference annotations, capturing specific evaluation signals without exposing raw data. To combine these heterogeneous supervision signals, MoR introduces a Mixture-of-Rewards mechanism with learned routing, which adaptively fuses client reward models according to the input and alignment objective. The server then optimizes a base VLM using GRPO with a KL penalty to a reference model, enabling preference alignment without requiring client models to share architectures or parameters. Experiments on diverse public vision-language benchmarks demonstrate that MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability. Our approach provides a scalable solution for privacy-preserving alignment of heterogeneous VLMs under federated settings.
BibTeX
@misc{lu2026replacing,
  author = {Shule Lu and Yujing Wang and Hainan Zhang and Xiaoshan Yang and Hongwei Zheng and Yongxin Tong and Changsheng Xu and Zhiming Zheng},
  title = {{Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models}},
  year = {2026},
  eprint = {2605.03426},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.03426},
}
Figure