Introduction
The Segment Anything Model (SAM) has been hailed as the "GPT moment" for computer vision, enabling precise object segmentation through simple prompts such as clicks or bounding boxes. Despite its power, SAM's "heavyweight" nature makes it difficult to deploy on resource-constrained edge devices like the NVIDIA Jetson Nano. The primary challenges are two-fold: the high computational demand of the image encoder and the latency associated with uploading high-resolution video to the cloud for remote processing.
The SAMEdge Framework
SAMEdge introduces a hybrid edge-cloud architecture designed to optimize SAM for real-time video analytics. It moves away from simple model compression, which often degrades accuracy, and instead focuses on intelligent workload distribution and prompt management.
Intelligent Visual Prompt Transformation
A significant bottleneck in SAM is the embedding decoder, which must execute for every visual prompt provided. SAMEdge addresses this by implementing a visual prompt transformation layer. Using an information-theoretical approach, it calculates a "visual prompt contribution score" based on mutual information. This allows the system to select and combine the most informative prompts, significantly reducing the decoder's workload without sacrificing segmentation quality.
Dynamic Workload Partitioning
Rather than a fixed offloading scheme, SAMEdge treats the neural network as a Directed Acyclic Graph (DAG). It utilizes a min-cut algorithm to identify the optimal splitting point between edge and cloud resources. This dynamic partitioning, known as the SAMEdge Resource Optimization (SERO) problem, is NP-hard, but the authors employ efficient solvers to handle real-time partitioning. By transmitting intermediate embeddings instead of raw video frames, SAMEdge reduces bandwidth requirements while leveraging cloud compute for the most intensive tasks.
Experimental Results
The researchers evaluated SAMEdge using the KITTI street-view dataset and the LIV360 dataset. The implementation utilized a TinyViT model as an efficient image prompt encoder to maintain a small edge footprint.
- Performance Gains: SAMEdge achieved a 4.64x improvement in the Number of Clicks (NoC) required to reach a 90% IoU on the KITTI dataset compared to vanilla edge deployments.
- Network Robustness: In scenarios with limited bandwidth (3 Mbps), where standard SAM performance degrades significantly, SAMEdge remained stable and accurate.
- Real-World Application: The system's utility was demonstrated through a "Visual Tour Guide" application, enabling real-time object identification with minimal user interaction (e.g., two clicks to identify a building).
- Zero-Shot Capability: Because SAMEdge does not modify the underlying model weights through pruning or quantization, it fully retains SAM's zero-shot segmentation capabilities.
Conclusion
SAMEdge demonstrates that edge-cloud collaboration is a viable path for bringing large-scale vision models to the edge. By intelligently managing prompts and partitioning workloads, it enables high-accuracy, interactive video analytics. Future research will focus on optimizing the heuristic solvers for resource allocation and exploring the privacy aspects of embedding-based communication.
Summary updated