Abstract
Video Object Segmentation (VOS) is a critical component for mobile applications ranging from augmented reality to advanced video editing. However, state-of-the-art VOS models typically rely on memory-intensive matching mechanisms that are unsuitable for mobile hardware. This paper introduces MobileVOS, a framework that unifies knowledge distillation (KD) and supervised contrastive learning (SCL) into a single representation loss. By leveraging a high-performance teacher model to guide a lightweight student, MobileVOS achieves a 32x reduction in parameters and a 5x speedup over existing efficient models, maintaining competitive accuracy while running at 30+ FPS on mobile devices.
Introduction
Video Object Segmentation (VOS) involves tracking and segmenting specific objects throughout a video sequence, typically initialized by a single mask in the first frame. While the field has seen significant progress through the use of "memory-based matching"—where features from all previous frames are stored in a memory bank—this approach presents a significant bottleneck for mobile deployment. The computational and memory requirements of these "infinite memory" models scale with video length, quickly exceeding the capabilities of smartphone processors.
Existing attempts to mitigate this through "finite memory" models (which store only a limited number of frames) often suffer from a substantial drop in segmentation quality. MobileVOS addresses this gap by introducing a training paradigm that allows a lightweight model to approximate the performance of high-end, memory-heavy architectures.
Methodology
Unified Representation Loss
The core innovation of MobileVOS is a novel loss function that mathematically unifies Knowledge Distillation (KD) and Supervised Contrastive Learning (SCL). The framework employs a large, high-performance teacher model (STCN) to guide a student model based on efficient backbones like MobileNetV2 or ResNet18.
- Knowledge Distillation (KD): This component transfers structural information from the teacher's feature maps to the student, allowing the smaller model to mimic the teacher's representational space.
- Supervised Contrastive Learning (SCL): Using ground-truth labels, this ensures that pixels belonging to the same object are pulled together in the feature space, while pixels from different objects are pushed apart.
The authors demonstrate that their representation loss is equivalent to maximizing the mutual information between the student and teacher. By using a hyperparameter ($\omega$), the model can interpolate between these two strategies during training, effectively treating SCL as a special case of distillation where the teacher is replaced by ground-truth labels.
Boundary-Aware Sampling
To optimize training, the researchers implemented boundary-aware sampling. Recognizing that the majority of segmentation errors occur at object edges, the training process prioritizes sampling pixels around these boundaries. This focus improves the model's ability to handle fine details and significantly accelerates the training convergence compared to uniform pixel sampling.
Finite Memory Management
Unlike the teacher model, which may store an extensive history of frames, the MobileVOS student utilizes a strict finite memory queue. It retains only two frames: the initial reference frame (to prevent drift from the original target) and the most recently processed frame. This fixed footprint ensures that memory usage remains constant regardless of video duration.
Experimental Setup
MobileVOS was evaluated on industry-standard benchmarks, including DAVIS 2016, DAVIS 2017, and YouTube-VOS 2019. The latter provides a large-scale test of the model's ability to generalize to over 65 unseen object categories. Models were pre-trained on static images with synthetic deformations to establish basic shape recognition before moving to video-based training.
Results and Discussion
Efficiency and Speed
MobileVOS demonstrates state-of-the-art efficiency for mobile VOS tasks. On a Samsung Galaxy S22, the model achieves real-time performance at over 30 FPS (approximately 32ms per frame). Compared to RDE-VOS, a competing efficient method, MobileVOS is 5x faster and utilizes 32x fewer parameters.
Accuracy and Robustness
Despite its small size, MobileVOS maintains high fidelity. On the DAVIS 2016 benchmark, it achieved a J&F score of 91.4, trailing the massive STCN teacher model by only 0.3 points. Qualitative analysis shows that the model is remarkably robust to complex occlusions and camera shot changes, successfully tracking objects where other finite-memory models fail.
Performance Optimization via Model Soups
The researchers found that averaging the weights of multiple models trained with different interpolation values ($\omega$) provided a "free" performance boost. This "model soup" approach improves accuracy without adding any computational overhead during inference.
Conclusion
MobileVOS proves that high-end video segmentation can be brought to mobile devices without sacrificing significant accuracy. By focusing on a superior training objective—unifying distillation and contrastive learning—rather than complex architectural changes, the authors have created a highly practical framework for real-time applications. Future work will explore applying this unified loss to other dense prediction tasks, such as real-time depth estimation and optical flow, while addressing the inherent challenges of drift in extremely long video sequences.
Summary updated