F3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics.

Pramit Saha, Felix Wagner, Divyanshu Mishra, Can Peng, Anshul Thakur, David A. Clifton, Konstantinos Kamnitsas, J. Alison Noble

Introduction

Fine-tuning massive Vision-Language Models (VLMs) like LLaVA or BLIP-2 is computationally prohibitive for resource-constrained devices. In Federated Learning (FL) environments—such as hospital networks or mobile fleets—local devices often lack the memory and compute power to update billions of parameters. While Parameter-Efficient Fine-Tuning (PEFT) offers a partial solution by updating only subsets of layers, current selection methods are often "naive," picking layers randomly or optimizing solely for individual clients. This leads to training inefficiency and a lack of diversity in the global model, where certain parameters remain "dead" or unoptimized.

To address these challenges, the authors introduce F³OCUS, a framework that optimizes layer selection by balancing local importance with global diversity.

The F³OCUS Framework

F³OCUS employs a two-step "define and refine" strategy to select which layers each client should update:

  1. Client-Level Importance: Each client calculates a "layer importance score" using the Layerwise Neural Tangent Kernel (LNTK). By analyzing the principal eigenvalue of the LNTK for each layer, clients identify which parameters align best with their local data distribution and will contribute most to rapid loss reduction.
  2. Server-Level Diversity: To prevent the global model from stagnating, the server performs multi-objective meta-heuristic optimization (utilizing algorithms like Genetic Algorithms or Simulated Annealing). This step maximizes total importance across the fleet while minimizing selection variance, ensuring a diverse set of layers is updated across the entire federation.
The F³OCUS architecture (bottom) compared to existing layer selection methods (top), highlighting the server-side multi-objective optimization for layer diversity.

Experimental Scope and Datasets

The study is massive in scale, involving over 10,000 client-level experiments. The authors released Ultra-MedVQA, the largest medical Visual Question Answering dataset to date, featuring over 700,000 VQA triplets across 9 modalities. Evaluations were conducted across multiple benchmarks (SLAKE, VQA-RAD, VQA-Med) using state-of-the-art models including ViLT, ALBEF, LLaVA-1.5, and BLIP-2.

Key Results and Insights

F³OCUS demonstrates significant improvements in both convergence speed and final accuracy compared to standard pruning and selection methods.

  • Superior Accuracy: F³OCUS outperformed existing methods like FishMask and SNIP by approximately 5.3% in both homogeneous and heterogeneous settings.
  • Faster Convergence: As shown in the training curves, F³OCUS reaches lower loss levels significantly faster than methods relying on weight magnitude or gradient norms alone.
  • Efficiency: The framework achieved high accuracy while maintaining a minimal communication footprint of just 9.7 MBits and reducing computational requirements to 80.6 GFLOPs.
Convergence comparison showing F³OCUS (blue) achieving significantly faster and deeper loss reduction compared to baseline selection methods like SNIP, GraSP, and NTK.

Conclusion and Future Work

F³OCUS proves that the principal eigenvalue of a layer's NTK is a principled metric for prioritization, but also highlights that global diversity is essential for federated success. By forcing the federation to "cover more ground" through meta-heuristic optimization, the model avoids the pitfalls of homogeneous layer selection.

Future research may focus on reducing the computational overhead of the server-side meta-heuristics as client counts scale, and developing dynamic adjustment strategies to handle real-time changes in client resource availability.

Summary updated

BibTeX
@inproceedings{saha2025f,
  author = {Pramit Saha and Felix Wagner and Divyanshu Mishra and Can Peng and Anshul Thakur and David A. Clifton and Konstantinos Kamnitsas and J. Alison Noble},
  title = {{F3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics.}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01863},
}
Figure