Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation

Guang-Yu Sun, Jingtao Li, Weiming Zhuang, Chen Chen 0043, Chen Chen 0001, Lingjuan LyuarXiv 2025 · DOI · arXiv

Introduction

Adapting large Foundation Models (FMs) like DINOv2 or ViT to private edge devices is a critical challenge for modern AI deployment. While these models offer immense power for downstream tasks such as autonomous driving, they are often too large for edge hardware and require labeled data that is rarely available in decentralized settings. This paper introduces FedMox, a framework designed to bridge the gap between high-performance server-side training and resource-constrained, unlabeled client environments. By using a sparse Mixture-of-Experts (MoE) architecture and a "Soft-Mixture" strategy, FedMox enables high-performance object detection on hardware as modest as a 2GB Raspberry Pi.

The Practical Semi-Supervised Federated Learning (PSSFL) Problem

Standard Federated Learning (FL) often fails in real-world scenarios due to three primary "walls":

  1. Computational Constraints: Training full FMs requires VRAM that far exceeds the capacity of typical edge devices.
  2. Label Scarcity: Most edge data is unlabeled, creating a semi-supervised learning challenge.
  3. Resolution Mismatch: To save memory, edge devices often process images at lower resolutions (e.g., 640x360) than the server (e.g., 1280x720), leading to a significant "feature gap."

The authors define this realistic environment as Practical Semi-Supervised Federated Learning (PSSFL).

Comparison of conventional FL and the PSSFL setting with memory requirements.

The FedMox Framework

To navigate these constraints, the authors propose FedMox (Federated Mixture of Experts). The framework employs a frozen FM backbone and focuses adaptation on the task head, significantly reducing the memory footprint.

Spatial Sparse Mixture-of-Experts

A single task head often lacks the capacity to learn from diverse, unlabeled data when the backbone is frozen. FedMox addresses this by using multiple "experts" within the task head. The key innovation is the spatial router, a 1x1 convolution that routes each pixel of a feature map to a specific expert.

The Spatial Routing mechanism for resolution-agnostic adaptation.

By treating pixels as individual tokens (similar to NLP MoE), the model maintains spatial consistency across resolutions. An expert trained on high-resolution features can effectively process low-resolution features because the router identifies local channel signatures rather than global geometry. Furthermore, the use of top-1 sparse activation ensures that only one expert is triggered per pixel, keeping computational overhead minimal (adding less than 1% to the detection head's FLOPs).

Soft-Mixture Strategy

In semi-supervised FL, sequential updates between the server (labeled data) and clients (unlabeled data) can lead to training fluctuations. FedMox introduces a Soft-Mixture strategy, which performs a weighted averaging of the previous server model and the new aggregated client model. This stabilizes convergence and balances the model's perception accuracy with its generalization capabilities.

Experimental Evaluation

FedMox was evaluated on three major autonomous driving benchmarks: BDD100K, SODA10M, and Cityscapes.

Key Results

  • Superior Accuracy: On the BDD100K dataset, FedMox achieved a total mAP@50 of 0.475, significantly outperforming baselines like FedAvg (0.448) and specialized semi-supervised methods like FedSTO (0.445).
  • Memory Efficiency: By combining a frozen backbone with low-resolution client training, FedMox reduced required client memory from 8.89GB to just 1.25GB, making it compatible with a 2GB Raspberry Pi.
  • Robustness and Scalability: The model maintained its performance edge when scaled to 100 clients and showed a 4% mAP improvement in domain generalization tests on the Cityscapes dataset, proving that the MoE structure helps the model adapt to unseen environments more effectively than a single task head.

Conclusion

FedMox provides a practical path for adapting Foundation Models to the edge. By addressing memory constraints, label scarcity, and resolution mismatches through a sparse MoE architecture and a stabilized mixture strategy, it enables high-performance object detection on resource-constrained hardware. Future work will explore extending this framework to other complex tasks like semantic segmentation and multi-modal adaptation.

Summary updated

Abstract

Foundation models (FMs) exhibit remarkable generalization but require adaptation to downstream tasks, particularly in privacy-sensitive applications. Due to data privacy regulations, cloud-based FMs cannot directly access private edge data, limiting their adaptation. Federated learning (FL) provides a privacy-aware alternative, but existing FL approaches overlook the constraints imposed by edge devices -- namely, limited computational resources and the scarcity of labeled data. To address these challenges, we introduce Practical Semi-Supervised Federated Learning (PSSFL), where edge devices hold only unlabeled, low-resolution data, while the server has limited labeled, high-resolution data. In this setting, we propose the Federated Mixture of Experts (FedMox), a novel framework that enhances FM adaptation in FL. FedMox tackles computational and resolution mismatch challenges via a sparse Mixture-of-Experts architecture, employing a spatial router to align features across resolutions and a Soft-Mixture strategy to stabilize semi-supervised learning. We take object detection as a case study, and experiments on real-world autonomous driving datasets demonstrate that FedMox effectively adapts FMs under PSSFL, significantly improving performance with constrained memory costs on edge devices. Our work paves the way for scalable and privacy-preserving FM adaptation in federated scenarios.
BibTeX
@misc{sun2025closer,
  author = {Guang-Yu Sun and Jingtao Li and Weiming Zhuang and Chen Chen and Chen Chen and Lingjuan Lyu},
  title = {{Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation}},
  year = {2025},
  eprint = {2508.16568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16568},
}
Figure