FOCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics

Pramit Saha, Felix Wagner, Divyanshu Mishra, Can Peng, Anshul Thakur, David A. Clifton, Konstantinos Kamnitsas, J. Alison Noble

TL;DR

This paper introduces FOCUS, a novel federated learning strategy for efficiently fine-tuning large Vision-Language Models (VLMs) on resource-constrained devices. FOCUS optimizes layer selection by balancing individual client needs with inter-client diversity using a multi-objective meta-heuristic approach, leading to improved performance and faster convergence.

What Problem Does This Paper Address?

Training massive Vision-Language Models (VLMs) on edge devices presents a significant challenge. These models, with billions of parameters, require substantial computational power and data, which are often limited on client devices. Federated Learning (FL) offers a privacy-preserving solution by training models collaboratively across these devices without centralizing data. However, adapting VLMs in FL settings is tricky due to:

  • Resource Constraints: Client devices have limited computational power and memory.
  • Data Heterogeneity: Data across clients can vary in distribution, modality, and task.
  • Parameter-Efficient Fine-Tuning (PEFT): Standard fine-tuning is too demanding, so PEFT methods are needed to update only a subset of parameters.
  • Suboptimal Layer Selection: Existing PEFT methods in FL often focus on local client data, ignoring the global learning dynamics and the need for diverse layer selection across clients, which can hinder overall convergence and performance.

This paper aims to bridge this gap by proposing a method that intelligently selects which layers of a VLM to fine-tune on each client, considering both the client's specific data and the collective learning needs of all clients.

Methods

The core of FOCUS lies in a two-step strategy executed at the beginning of each training round:

  1. Client-Level Layer Importance Score: Each client calculates an "importance score" for each VLM layer. This score is derived from the principal eigenvalue magnitude of the Layerwise Neural Tangent Kernel (LNTK). The LNTK captures how well a layer's parameters align with the client's data distribution, indicating its potential contribution to learning. This is a novel application of LNTK for layer selection in FL.
  2. Server-Level Multi-Objective Meta-Heuristic Optimization: The server aggregates these client-specific importance scores. It then employs a data-free, multi-objective meta-heuristic optimization to refine the layer selection. The objectives are:
  3. Maximize cumulative client-specific importance: Prioritize layers that are important across many clients.
  4. Minimize the variance of layer selection histogram: Encourage diverse layer selection across clients, preventing over-reliance on a few layers and promoting balanced learning.

    The paper explores five meta-heuristic algorithms (Genetic Algorithm, Artificial Bee Colony, Ant Colony Optimization, Simulated Annealing, and Swarm Particle Optimization) to find the optimal trade-off between these objectives. This server-side optimization ensures that the global learning process benefits from both individual client needs and collective diversity.

Datasets

The authors introduce a new benchmark dataset:

  • Ultra-MedVQA-FL: A large-scale medical Visual Question Answering (VQA) dataset comprising 707,962 VQA triplets. It features 9 modality-specific clients and 12 distinct anatomies, covering a wide range of medical imaging tasks.

They also utilize existing datasets for evaluation across various tasks, including:

  • Visual Question Answering (VQA): SLAKE, VQA-RAD, VQA-Med (2019, 2020, 2021).
  • Image and Text-based Disease Classification: Open-I, MIMIC.
  • Heterogeneous Tasks: Combining VQA and disease classification.

Key Insights and Ideas

  • LNTK for Layer Importance: Using the principal eigenvalue of the LNTK is a theoretically sound way to measure a layer's importance for a specific client's data, directly linking it to learning dynamics and convergence.
  • Client-Server Synergy: The proposed FOCUS strategy effectively combines client-specific insights (LNTK scores) with global optimization (server-side meta-heuristics) to achieve better results than methods that only consider one aspect.
  • Diversity is Key: Simply maximizing individual client importance isn't enough. Encouraging diversity in layer selection across clients is crucial for robust and efficient federated learning, especially in heterogeneous environments.
  • Meta-Heuristics for Complex Optimization: The problem of balancing multiple, conflicting objectives (importance vs. diversity) in layer selection is well-suited for meta-heuristic algorithms, which can explore complex solution spaces without requiring data on the server.

Key Results

  • Superior Performance: FOCUS consistently outperforms state-of-the-art PEFT methods and layer selection strategies across various Vision-Language FL tasks and VLM architectures. For example, in comparisons with other layer selection methods, FOCUS achieved improvements of up to 5.35% and 5.33% over the closest competitors.
  • Faster Convergence: Figure 2 demonstrates that FOCUS leads to faster loss convergence compared to client-specific NTK-based selection alone, highlighting the benefit of server-level optimization.
  • Effective Layer Selection: Visualizations (Figures 5, 6, and 7) show how FOCUS refines client-level layer ranks and promotes more balanced layer participation across clients. Figure 9 shows improved feature separability for a specific client using FOCUS.
  • Efficiency: FOCUS achieves these gains while fine-tuning fewer layers than methods that fine-tune all adapters, leading to reduced communication and computational costs.

Limitations and Future Work

  • Computational Cost of LNTK: Calculating LNTK can be computationally intensive, although the paper focuses on its application for layer selection rather than full model training.
  • Meta-Heuristic Tuning: The performance of the meta-heuristic optimization can depend on hyperparameter tuning.
  • Future Work: The authors suggest exploring more sophisticated meta-heuristic algorithms, investigating adaptive layer selection strategies that evolve over time, and extending the framework to other federated learning scenarios beyond VLM fine-tuning. The release of the Ultra-MedVQA-FL dataset also opens avenues for future research in medical VLM adaptation.

Summary updated

Abstract

Effective training of large Vision-Language Models (VLMs) on resource-constrained client devices in Federated Learning (FL) requires the usage of parameter-efficient fine-tuning (PEFT) strategies. To this end, we demonstrate the impact of two factors \textit{viz.}, client-specific layer importance score that selects the most important VLM layers for fine-tuning and inter-client layer diversity score that encourages diverse layer selection across clients for optimal VLM layer selection. We first theoretically motivate and leverage the principal eigenvalue magnitude of layerwise Neural Tangent Kernels and show its effectiveness as client-specific layer importance score. Next, we propose a novel layer updating strategy dubbed F$^3$OCUS that jointly optimizes the layer importance and diversity factors by employing a data-free, multi-objective, meta-heuristic optimization on the server. We explore 5 different meta-heuristic algorithms and compare their effectiveness for selecting model layers and adapter layers towards PEFT-FL. Furthermore, we release a new MedVQA-FL dataset involving overall 707,962 VQA triplets and 9 modality-specific clients and utilize it to train and evaluate our method. Overall, we conduct more than 10,000 client-level experiments on 6 Vision-Language FL task settings involving 58 medical image datasets and 4 different VLM architectures of varying sizes to demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{saha2025focus,
  author = {Pramit Saha and Felix Wagner and Divyanshu Mishra and Can Peng and Anshul Thakur and David A. Clifton and Konstantinos Kamnitsas and J. Alison Noble},
  title = {{FOCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01863},
}
Figure