Introduction
Adapting large Foundation Models (FMs) like DINOv2 or ViT to private edge devices is a critical challenge for modern AI deployment. While these models offer immense power for downstream tasks such as autonomous driving, they are often too large for edge hardware and require labeled data that is rarely available in decentralized settings. This paper introduces FedMox, a framework designed to bridge the gap between high-performance server-side training and resource-constrained, unlabeled client environments. By using a sparse Mixture-of-Experts (MoE) architecture and a "Soft-Mixture" strategy, FedMox enables high-performance object detection on hardware as modest as a 2GB Raspberry Pi.
The Practical Semi-Supervised Federated Learning (PSSFL) Problem
Standard Federated Learning (FL) often fails in real-world scenarios due to three primary "walls":
- Computational Constraints: Training full FMs requires VRAM that far exceeds the capacity of typical edge devices.
- Label Scarcity: Most edge data is unlabeled, creating a semi-supervised learning challenge.
- Resolution Mismatch: To save memory, edge devices often process images at lower resolutions (e.g., 640x360) than the server (e.g., 1280x720), leading to a significant "feature gap."
The authors define this realistic environment as Practical Semi-Supervised Federated Learning (PSSFL).
The FedMox Framework
To navigate these constraints, the authors propose FedMox (Federated Mixture of Experts). The framework employs a frozen FM backbone and focuses adaptation on the task head, significantly reducing the memory footprint.
Spatial Sparse Mixture-of-Experts
A single task head often lacks the capacity to learn from diverse, unlabeled data when the backbone is frozen. FedMox addresses this by using multiple "experts" within the task head. The key innovation is the spatial router, a 1x1 convolution that routes each pixel of a feature map to a specific expert.
By treating pixels as individual tokens (similar to NLP MoE), the model maintains spatial consistency across resolutions. An expert trained on high-resolution features can effectively process low-resolution features because the router identifies local channel signatures rather than global geometry. Furthermore, the use of top-1 sparse activation ensures that only one expert is triggered per pixel, keeping computational overhead minimal (adding less than 1% to the detection head's FLOPs).
Soft-Mixture Strategy
In semi-supervised FL, sequential updates between the server (labeled data) and clients (unlabeled data) can lead to training fluctuations. FedMox introduces a Soft-Mixture strategy, which performs a weighted averaging of the previous server model and the new aggregated client model. This stabilizes convergence and balances the model's perception accuracy with its generalization capabilities.
Experimental Evaluation
FedMox was evaluated on three major autonomous driving benchmarks: BDD100K, SODA10M, and Cityscapes.
Key Results
- Superior Accuracy: On the BDD100K dataset, FedMox achieved a total mAP@50 of 0.475, significantly outperforming baselines like FedAvg (0.448) and specialized semi-supervised methods like FedSTO (0.445).
- Memory Efficiency: By combining a frozen backbone with low-resolution client training, FedMox reduced required client memory from 8.89GB to just 1.25GB, making it compatible with a 2GB Raspberry Pi.
- Robustness and Scalability: The model maintained its performance edge when scaled to 100 clients and showed a 4% mAP improvement in domain generalization tests on the Cityscapes dataset, proving that the MoE structure helps the model adapt to unseen environments more effectively than a single task head.
Conclusion
FedMox provides a practical path for adapting Foundation Models to the edge. By addressing memory constraints, label scarcity, and resolution mismatches through a sparse MoE architecture and a stabilized mixture strategy, it enables high-performance object detection on resource-constrained hardware. Future work will explore extending this framework to other complex tasks like semantic segmentation and multi-modal adaptation.
Summary updated