Abstract
Camouflaged object segmentation (COS) presents unique challenges due to the high similarity in patterns and colors between objects and their backgrounds. While prior research has emphasized supervised pre-training, zero-shot approaches have remained underdeveloped and underperforming. We introduce ZS-VCOS, a modular zero-shot pipeline that integrates optical flow, vision-language models (VLMs), and SAM 2. Evaluated on the MoCA-Mask dataset, our approach significantly outperforms existing zero-shot methods, raising the weighted F-measure ($F_\beta^w$) from 0.296 to 0.628. Remarkably, ZS-VCOS also surpasses state-of-the-art supervised methods, which previously peaked at 0.476. Further evaluation on the MoCA-Filter dataset demonstrates a success rate increase to 0.697, outperforming supervised transfer methods like FlowSAM.
Introduction
Camouflaged Object Segmentation is often considered the "hard mode" of computer vision. Unlike standard segmentation tasks where objects are distinct from their surroundings, camouflaged entities have evolved to blend in, sharing nearly identical textures and colors with their environments.
In video-based COS (VCOS), motion cues can be leveraged to break this camouflage. However, the field has been dominated by supervised learning models trained on massive, manually labeled datasets. Existing zero-shot attempts typically rely on visual features that are intentionally hidden, leading to poor performance. ZS-VCOS addresses this by building a modular pipeline that "sees" what even specialized supervised models often miss by combining generalist foundation models.
The ZS-VCOS Pipeline
The authors propose a sequential pipeline that bridges the gap between low-level motion cues and high-level semantic understanding:
- Motion Detection (The "Highlighter"): The system uses RAFT to compute dense optical flow. To handle camera movement, a Lucas-Kanade-based motion compensation technique is applied. The resulting motion is converted into an intensity map and "blended" into the original frame using a specific color (e.g., blue), physically altering the input to highlight moving areas.
- Open Vocabulary Detection: Instead of asking a VLM to find a hidden animal, the system uses Owl-v2 to find "an animal or insect being highlighted in blue." This transforms a difficult camouflage detection task into a simpler detection task.
- Segmentation and Tracking: The bounding box from the VLM and the center of mass from the motion map serve as prompts for SAM 2.
- Bidirectional Propagation: To maintain temporal consistency, tracking is run both forward and backward in time, with results combined via an OR operation.
Key Results
The performance gains achieved by ZS-VCOS are substantial across multiple benchmarks. On the MoCA-Mask dataset, ZS-VCOS achieved a weighted F-measure ($F_\beta^w$) of 0.628, more than doubling the previous zero-shot best (0.296) and significantly exceeding the supervised state-of-the-art, ZoomNeXt (0.476). In terms of mIoU, the method reached 0.550, compared to the 0.273 baseline of previous zero-shot methods.
On the MoCA-Filtered subset, which focuses on locomotive videos, ZS-VCOS achieved a detection success rate of 0.697, outperforming supervised transfer methods like FlowP-SAM (0.645).
Discussion and Conclusion
ZS-VCOS demonstrates that we have reached a tipping point where a pipeline of "generalist" foundation models can outperform "specialist" models trained for niche tasks. A key innovation is using motion as a visual prompt to guide VLMs, effectively bridging the gap between low-level temporal cues and high-level semantics.
While the system currently faces limitations in multi-object scenarios and with long-term stationary objects, its success highlights the importance of temporal consistency and the potential of zero-shot architectures in complex computer vision domains. Future work will explore automatic prompt generation and few-shot detection to further enhance autonomy and precision.
Summary updated