F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics

Pramit Saha, Felix Wagner, Divyanshu Mishra, Can Peng, Anshul Thakur, David A. Clifton, Konstantinos Kamnitsas, J. Alison Noble

TL;DR

This paper introduces F³OCUS, a novel federated learning strategy for fine-tuning large Vision-Language Models (VLMs) on resource-constrained devices. It tackles the challenge of balancing individual client needs with global model performance by optimizing both client-specific layer importance and inter-client layer diversity using a meta-heuristic approach on the server, leading to improved accuracy and convergence.

What Problem Does This Paper Address?

Training massive Vision-Language Models (VLMs) for specialized tasks often requires fine-tuning them on domain-specific data. However, in scenarios like healthcare, data privacy regulations prevent central aggregation, making Federated Learning (FL) a more suitable approach. FL allows models to be trained on local devices without sharing sensitive data. The challenge, however, is that fine-tuning large VLMs locally is computationally demanding and often infeasible due to limited client resources. Parameter-Efficient Fine-Tuning (PEFT) methods offer a solution by only updating a subset of model parameters. Yet, existing PEFT strategies in FL often overlook the diverse computational capabilities and data distributions across clients, leading to suboptimal performance and slow convergence. This paper addresses this gap by proposing a method that intelligently selects which VLM layers to fine-tune on each client, considering both the client's specific needs and the collective benefit of the entire federated network.

Methods

The core of F³OCUS lies in a two-step strategy:

  1. Client-Level Layer Importance: Clients determine the importance of each VLM layer for their local data. This is achieved by leveraging the Layerwise Neural Tangent Kernel (LNTK). Specifically, they compute the principal eigenvalue of the LNTK for each layer. The intuition here is that layers with higher principal eigenvalues are more aligned with the client's data distribution and thus contribute more significantly to learning.
  2. Server-Level Multi-Objective Meta-Heuristic Optimization: The server receives these client-specific importance scores. Instead of simply averaging them, F³OCUS employs a data-free, multi-objective, meta-heuristic optimization to refine the layer selection. The server aims to achieve two primary objectives:
  3. Maximize cumulative client-specific importance: Ensure that the selected layers are generally important across many clients.
  4. Minimize the variance of the layer selection histogram: Encourage diversity in layer selection across clients. This prevents over-reliance on a few highly important layers and promotes more balanced participation of all layers, leading to better generalization and robustness.

    The paper explores five different meta-heuristic algorithms (NSGA, ABC, ACO, SA, MOPSO) for this server-side optimization, allowing for flexibility in choosing the best approach.

Datasets

The authors introduce a new dataset called Ultra-MedVQA.

  • Ultra-MedVQA: This is presented as the largest medical VQA dataset to date, containing over 707,962 VQA triplets. It spans 9 different modalities and 12 distinct anatomies, covering a wide range of medical imaging tasks.
  • Existing Datasets: The experiments also utilize several other medical VQA datasets, including SLAKE, VQA-RAD, VQA-Med 2019, VQA-Med 2020, and VQA-Med 2021, as well as datasets for image and text-based multi-label disease classification (Open-I, MIMIC).

Key Insights and Ideas

  • LNTK as a Proxy for Layer Importance: The paper theoretically motivates and empirically demonstrates that the principal eigenvalue of the LNTK is a strong indicator of a layer's importance for a specific client's data. This provides a principled way to quantify layer relevance without needing direct access to client data.
  • The Power of Server-Side Refinement: Simply aggregating client-specific layer importance scores is insufficient. The server's role in optimizing for both cumulative importance and inter-client diversity is crucial for achieving superior performance and faster convergence.
  • Meta-Heuristics for Complex Optimization: The problem of balancing multiple, potentially conflicting objectives (layer importance vs. diversity) in a data-scarce server environment is well-suited for meta-heuristic optimization. The paper shows that various meta-heuristics can effectively tackle this challenge.
  • Diversity is Key for Robustness: Encouraging diverse layer selection across clients, rather than having all clients focus on the same few "most important" layers, leads to a more robust and better-performing federated model.

Key Results

  • Improved Accuracy: F³OCUS consistently outperforms various state-of-the-art PEFT methods and federated learning baselines across multiple tasks and VLM architectures. For instance, in Task 2 with heterogeneous resources, F³OCUS achieved an accuracy of 84.25% (VILT) and 80.46% (BLIP), significantly outperforming other methods.
  • Faster Convergence: The paper shows that F³OCUS leads to faster convergence compared to other methods, as visualized in Figure 2.
  • Reduced Communication/Computation: By intelligently selecting layers, F³OCUS achieves competitive performance with significantly fewer trainable parameters and lower communication overhead compared to methods that fine-tune all adapters.
  • Effective Meta-Heuristic Performance: The paper compares five meta-heuristic algorithms, showing that they all perform comparably well, with F³OCUS consistently outperforming baselines regardless of the chosen meta-heuristic.
  • Visualizations Highlight Improvement: t-SNE visualizations (Figure 9) demonstrate that F³OCUS generates more distinctive feature representations compared to random selection or LNTK-based selection alone.

Limitations and Future Work

  • Computational Cost of LNTK: While effective, computing the LNTK can still be computationally intensive for clients, especially for very large models.
  • Meta-Heuristic Selection: The paper explores several meta-heuristics but doesn't definitively declare one as universally superior. Future work could involve more in-depth analysis or adaptive selection of meta-heuristics.
  • Generalization to Other Tasks: While F³OCUS is demonstrated on VLM tasks, its applicability to other domains of federated learning could be explored.
  • Dynamic Client Participation: The current framework assumes clients participate consistently. Future work could investigate how F³OCUS handles scenarios with highly dynamic client availability.

Summary updated

Abstract

Effective training of large Vision-Language Models (VLMs) on resource-constrained client devices in Federated Learning (FL) requires the usage of parameter-efficient fine-tuning (PEFT) strategies. To this end, we demonstrate the impact of two factors, viz., client-specific layer importance score that selects the most important VLM layers for fine-tuning and inter-client layer diversity score that encourages diverse layer selection across clients for optimal VLM layer selection. We first theoretically motivate and leverage the principal eigenvalue magnitude of layerwise Neural Tangent Kernels and show its effectiveness as client-specific layer importance score. Next, we propose a novel layer updating strategy dubbed F^3OCUS that jointly optimizes the layer importance and diversity factors by employing a data-free, multi-objective, meta-heuristic optimization on the server. We explore 5 different meta-heuristic algorithms and compare their effectiveness for selecting model layers and adapter layers towards PEFT-FL. Furthermore, we release a new MedVQA-FL dataset involving overall 707,962 VQA triplets and 9 modality-specific clients and utilize it to train and evaluate our method. Overall, we conduct more than 10,000 client-level experiments on 6 Vision-Language FL task settings involving 58 medical image datasets and 4 different VLM architectures of varying sizes to demonstrate the effectiveness of the proposed method. Project Page: https://pramitsaha.github.io/FOCUS/
BibTeX
@inproceedings{saha2025f,
  author = {Pramit Saha and Felix Wagner and Divyanshu Mishra and Can Peng and Anshul Thakur and David A. Clifton and Konstantinos Kamnitsas and J. Alison Noble},
  title = {{F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01863},
}
Figure