Introduction
The Segment Anything Model 2 (SAM2) represents a significant leap in image and video segmentation. However, its massive memory footprint and computational intensity pose substantial barriers to deployment on resource-constrained edge devices, such as mobile phones and embedded cameras. While previous iterations like MobileSAM and TinySAM addressed image-level efficiency, video segmentation introduces the unique challenge of temporal redundancy. Standard models often treat consecutive, highly similar frames as independent problems, leading to significant computational waste.
To address these challenges, researchers from Xidian University have introduced Q-MiniSAM2, the first quantization-based benchmark designed to optimize SAM2 for edge environments.
Methodology
The Q-MiniSAM2 framework employs a three-pronged strategy to reduce model size and latency while preserving segmentation accuracy. A key insight driving this approach is the "97% Rule": the authors identified that just three modules—the image encoder, memory attention, and mask decoder—account for over 97% of SAM2's parameters.
1. Post-Training Quantization (PTQ)
Rather than undergoing the computationally expensive process of Quantization-Aware Training (QAT), the authors utilize PTQ. This approach converts high-precision floating-point weights and activations into low-bit integers (e.g., 6-bit or 4-bit) using a small, unlabeled calibration dataset.
2. Hierarchy-based Video Quantization (HVQ)
To mitigate the performance degradation typically associated with low-bit quantization, HVQ leverages contrastive learning. It treats the current frame as an "anchor" and optimizes the model to maintain consistency across temporal frames (nearby frames with high visual similarity) and semantic frames (distant frames providing broader context).
3. Adaptive Mutual-KV Mechanism
This mechanism targets the redundancy in SAM2's cross-attention layers. By calculating frame similarity, the model can share "Key-Value" (KV) pairs between similar historical frames, drastically reducing memory overhead and redundant computations.
Experimental Results
The model was evaluated on the MS-COCO dataset for image segmentation and the SA-V dataset—the largest available video segmentation benchmark.
Efficiency and Performance
Q-MiniSAM2 achieves a remarkable balance between efficiency and accuracy. At 6-bit precision (W6A6), the model retains 93% of the full-precision SAM2 performance while reducing storage requirements by 80%. In more aggressive 4-bit (W4A4) configurations, Q-MiniSAM2 significantly outperforms existing methods like PTQ4SAM on the SA-V benchmark in both Jaccard Index ($\\mathcal{J}$) and F-measure ($\\mathcal{F}$).
Speedup
The reduction in bit-width translates to significant theoretical speedups. Using 4-bit arithmetic on specialized hardware can be up to 8x faster than standard 32-bit operations. The authors anticipate real-world inference speeds to be at least 5x faster than the original SAM2 model.
Conclusion
Q-MiniSAM2 demonstrates that high-quality video segmentation is achievable on edge hardware through strategic quantization and redundancy elimination. While 6-bit quantization offers a robust "sweet spot" for deployment, future work will focus on closing the performance gap for 4-bit quantization to enable even more aggressive model compression.
Summary updated