EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge

Bufang Yang, Lixing He, Neiwen Ling, Zhenyu Yan, Guoliang Xing, Xian Shuai, Xiaozhe Ren, Xin Jiang

Introduction

Deep learning on edge devices typically relies on small, task-specific models. While efficient, these models are often "closed-set," failing when encountering classes not seen during training, and lack generalizability across diverse environments. Conversely, large Foundation Models (FMs) like CLIP or ImageBind offer robust "open-set" recognition but are too computationally expensive for edge hardware. Streaming all data to the cloud is also impractical due to latency and bandwidth constraints.

EdgeFM addresses this by proposing an edge-cloud cooperative system. It treats cloud-based FMs as a "knowledge base" to continuously upgrade local, lightweight models, enabling open-set recognition on resource-constrained IoT devices.

System Architecture and Methods

EdgeFM utilizes a multi-pronged approach to bridge the gap between edge efficiency and FM intelligence.

EdgeFM system architecture showing the interaction between cloud-based Foundation Models and edge-based customized small models.

The system consists of several key components:

  1. Semantic-driven Customization: Instead of traditional knowledge distillation, EdgeFM aligns the features of a local CNN (e.g., MobileNet) with the FM’s unified embedding space. This enables the small model to perform open-set recognition by matching its feature outputs against text embeddings.
  2. Label-free Training: The system leverages FM predictions as "pseudo-labels" to train the local model on unlabeled data collected in situ.
  3. Dynamic Model Switching: A runtime "router" decides whether to use the local model or query the cloud FM based on uncertainty quantification and real-time network conditions.
  4. Content-aware Data Uploading: To optimize bandwidth, only samples with high uncertainty are uploaded for further refinement.

Datasets and Evaluation

The system was evaluated on five datasets covering vision and audio modalities:

  • Vision: FLO102 (flower recognition) and UCF101 (human activity).
  • Audio: ESC50 (environmental sounds).
  • Self-collected: SC15 (real-world home activity) and SC40 (indoor objects for mobile robots), designed specifically for dynamic open-set scenarios.

Analysis and Key Insights

A core finding is the feasibility of using FMs to supervise small models. As the accuracy of the FM supervision increases, the performance of the customized small model follows closely, demonstrating that FMs can effectively serve as a knowledge base for edge adaptation.

Feasibility analysis showing that the accuracy of the small model scales linearly with the accuracy of the FM-provided supervision.

Furthermore, the study reveals that:

  • Specialization: Customized small models can eventually outperform the original FM on specific tasks by specializing in the local data distribution.
  • Embedding-based Open-set: By matching FM embeddings, small models inherit open-set capabilities without the overhead of transformer architectures.
  • Network Adaptability: EdgeFM adjusts its query threshold based on bandwidth, maintaining responsiveness in fluctuating network environments.

Experimental Results

EdgeFM demonstrates significant improvements over traditional edge-only or cloud-only approaches:

  • Accuracy: Achieved a 34.3% increase over existing on-device open-set recognition baselines.
  • Latency: Reduced end-to-end latency by up to 3.2x compared to cloud-centric models.
  • Efficiency: Content-aware uploading reduced data transmission by 60% with minimal accuracy loss.
  • Robustness: Successfully maintained high accuracy in robot navigation tests with bandwidth ranging from 2 Mbps to 123 Mbps.

Conclusion and Future Work

EdgeFM successfully brings FM-level intelligence to the edge. Future research will explore integrating a wider variety of FMs (e.g., SAM for segmentation), addressing covariate shifts (like lighting changes), and extending the framework to other sensor modalities such as IMU data for health monitoring.

Summary updated

BibTeX
@inproceedings{yang2023edgefm,
  author = {Bufang Yang and Lixing He and Neiwen Ling and Zhenyu Yan and Guoliang Xing and Xian Shuai and Xiaozhe Ren and Xin Jiang},
  title = {{EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge}},
  booktitle = {SenSys},
  year = {2023},
  url = {https://openreview.net/forum?id=UzE5pjpHA0},
}

Loading related content...

Figure