TL;DR
MobileSAM2 brings Meta’s powerful Segment Anything Model 2 (SAM2) to resource-constrained edge devices by distilling it into a lightweight model family. By introducing Hypergraphical Knowledge Distillation (HyperKD), the authors successfully transfer complex temporal and multi-granularity relationships from the heavyweight teacher to a compact student. The resulting models achieve state-of-the-art mobile video segmentation performance and show strong generalization on embodied AI tasks.
What Problem Does This Paper Address?
Meta's SAM2 revolutionized computer vision by enabling promptable "segment anything" capabilities across both static images and video streams. However, this power comes at a steep computational cost: its default image encoder (Hiera-L) contains over 212 million parameters, making it a non-starter for real-time deployment on mobile phones, drones, or robotic platforms.
While knowledge distillation (KD) is a standard approach to compress large models into smaller "students," traditional KD methods struggle with video foundation models. Standard techniques typically align individual pixel features or simple pairwise relationships. They fail to capture the high-order, complex structures that make SAM2 successful—specifically, tracking objects consistently across multiple video frames (temporal knowledge) and understanding hierarchical boundaries like objects, parts, and subparts (multi-granularity knowledge).
Methods
To bridge this gap, the authors propose Hypergraphical Knowledge Distillation (HyperKD) and a new family of models called MobileSAM2.
1. Hypergraphical Knowledge Distillation (HyperKD)
Unlike standard graphs where an edge connects exactly two nodes, a hypergraph features hyperedges that can connect multiple nodes simultaneously. This makes them uniquely suited for modeling complex, multi-way interactions. HyperKD splits this task into two specialized components:
- Temporal HyperKD: This module extracts SAM2's frame-to-frame tracking capabilities. It treats image patches or object instances within a single frame as nodes and links related nodes across multiple frames using hyperedges, enforcing temporal consistency in the student model.
- Granularity HyperKD: This module captures hierarchical spatial relationships. It treats segmentation entities at different levels of detail (e.g., an entire car vs. its wheels vs. its lug nuts) as nodes and groups them via hyperedges to teach the student coarse-to-fine spatial reasoning.
2. Progressive Model Contraction
To find the optimal lightweight architecture, the authors use a progressive model contraction strategy. They iteratively shrink a baseline Hiera vision transformer along four structural axes: embedding dimensions, layer counts, window sizes, and attention head configurations. Instead of guessing which architecture is best, they use the HyperKD loss to identify the configurations that retain the most knowledge during size reduction.
Datasets
The models were trained and evaluated using several prominent video object segmentation (VOS) benchmarks:
- SA-V (Segment Anything in Videos): The primary dataset used for distillation. Impressively, the authors only used a ~10% subset (11K videos) for training, demonstrating high data efficiency.
- MOSE & DAVIS 2017: Standard benchmarks used to evaluate general video object segmentation in complex scenes.
- LVOS: A long-term video object segmentation dataset used to test long-range temporal consistency.
- LIBERO-PRO: A robotic manipulation benchmark used to evaluate MobileSAM2's performance as a perception engine for embodied AI agents.
Key Insights and Ideas
- High-Order Structures Matter: Traditional pairwise relational distillation is insufficient for video foundation models. Capturing multi-node interactions via hypergraphs prevents the student model from losing tracking coherence over time.
- Task Losses Can Be Counterproductive: Counterintuitively, the authors found that removing task-specific segmentation losses (like cross-entropy) during distillation improved performance by +1.8 J&F points. When training data is limited (e.g., the 10% SA-V subset), standard task losses can introduce noise and conflict with the rich structural guidance provided by the teacher hypergraph.
- Cross-Model Applicability: HyperKD is not strictly bound to the SAM2 architecture. The authors successfully applied it to TrackAnything (TAM), an entirely independent model architecture, achieving a significant performance boost (+4.2 J&F on DAVIS-2017). This proves HyperKD transfers genuine semantic knowledge rather than just over-fitting to a specific backbone.
Key Results
The authors introduced three versions of MobileSAM2 (5M, 10M, and 23M parameters) and evaluated them against state-of-the-art lightweight baselines.
- Segmentation Accuracy: On the long-term LVOS dataset, MobileSAM2-23M achieved a J&F score of 69.0, vastly outperforming standard distillation methods like FitNet (60.8) and recent advanced baselines like EA-KD (63.6).
- Hardware Efficiency: While the original SAM2 Base and Large models suffered from "Out of Memory" errors on a mobile GPU (RTX 4060 Mobile), MobileSAM2-5M ran comfortably at 13.3 FPS while maintaining a highly competitive J&F score.
- Embodied AI: When integrated into a robotic agent (
CaP-Agent0), MobileSAM2-23M achieved a 21.8% average success rate on manipulation tasks, outperforming a state-of-the-art TinyViT baseline (19.3%) and approaching the performance of the massive, uncompressed teacher model (23.2%).
Limitations and Future Work
The primary limitation noted is that the progressive contraction strategy relies on a manually designed search space for the architecture axes (dimensions, blocks, etc.). Because it lacks a fully automated Neural Architecture Search (NAS) with optimality guarantees, it is possible that an even more efficient hardware-tailored architecture exists. The authors suggest that automating this search space is a promising direction for future work.
Summary updated