ZS-VCOS: Zero-Shot Outperforms Supervised Video Camouflaged Object Segmentation

Wenqi Guo, Shan Du

Abstract

Camouflaged object segmentation (COS) presents unique challenges due to the high similarity in patterns and colors between objects and their backgrounds. While prior research has emphasized supervised pre-training, zero-shot approaches have remained underdeveloped and underperforming. We introduce ZS-VCOS, a modular zero-shot pipeline that integrates optical flow, vision-language models (VLMs), and SAM 2. Evaluated on the MoCA-Mask dataset, our approach significantly outperforms existing zero-shot methods, raising the weighted F-measure ($F_\beta^w$) from 0.296 to 0.628. Remarkably, ZS-VCOS also surpasses state-of-the-art supervised methods, which previously peaked at 0.476. Further evaluation on the MoCA-Filter dataset demonstrates a success rate increase to 0.697, outperforming supervised transfer methods like FlowSAM.

Introduction

Camouflaged Object Segmentation is often considered the "hard mode" of computer vision. Unlike standard segmentation tasks where objects are distinct from their surroundings, camouflaged entities have evolved to blend in, sharing nearly identical textures and colors with their environments.

In video-based COS (VCOS), motion cues can be leveraged to break this camouflage. However, the field has been dominated by supervised learning models trained on massive, manually labeled datasets. Existing zero-shot attempts typically rely on visual features that are intentionally hidden, leading to poor performance. ZS-VCOS addresses this by building a modular pipeline that "sees" what even specialized supervised models often miss by combining generalist foundation models.

The ZS-VCOS Pipeline

The authors propose a sequential pipeline that bridges the gap between low-level motion cues and high-level semantic understanding:

  1. Motion Detection (The "Highlighter"): The system uses RAFT to compute dense optical flow. To handle camera movement, a Lucas-Kanade-based motion compensation technique is applied. The resulting motion is converted into an intensity map and "blended" into the original frame using a specific color (e.g., blue), physically altering the input to highlight moving areas.
  2. Open Vocabulary Detection: Instead of asking a VLM to find a hidden animal, the system uses Owl-v2 to find "an animal or insect being highlighted in blue." This transforms a difficult camouflage detection task into a simpler detection task.
  3. Segmentation and Tracking: The bounding box from the VLM and the center of mass from the motion map serve as prompts for SAM 2.
  4. Bidirectional Propagation: To maintain temporal consistency, tracking is run both forward and backward in time, with results combined via an OR operation.
The ZS-VCOS modular pipeline integrates motion detection, open-vocabulary detection, and SAM-2 tracking.

Key Results

The performance gains achieved by ZS-VCOS are substantial across multiple benchmarks. On the MoCA-Mask dataset, ZS-VCOS achieved a weighted F-measure ($F_\beta^w$) of 0.628, more than doubling the previous zero-shot best (0.296) and significantly exceeding the supervised state-of-the-art, ZoomNeXt (0.476). In terms of mIoU, the method reached 0.550, compared to the 0.273 baseline of previous zero-shot methods.

Evolution of $F_\beta^w$ score, showing ZS-VCOS significantly outperforming both previous zero-shot and supervised methods over time.

On the MoCA-Filtered subset, which focuses on locomotive videos, ZS-VCOS achieved a detection success rate of 0.697, outperforming supervised transfer methods like FlowP-SAM (0.645).

Discussion and Conclusion

ZS-VCOS demonstrates that we have reached a tipping point where a pipeline of "generalist" foundation models can outperform "specialist" models trained for niche tasks. A key innovation is using motion as a visual prompt to guide VLMs, effectively bridging the gap between low-level temporal cues and high-level semantics.

While the system currently faces limitations in multi-object scenarios and with long-term stationary objects, its success highlights the importance of temporal consistency and the potential of zero-shot architectures in complex computer vision domains. Future work will explore automatic prompt generation and few-shot detection to further enhance autonomy and precision.

Summary updated

Abstract

Camouflaged object segmentation presents unique challenges compared to traditional segmentation tasks, primarily due to the high similarity in patterns and colors between camouflaged objects and their backgrounds. Effective solutions to this problem have significant implications in critical areas such as pest control, defect detection, and lesion segmentation in medical imaging. Prior research has predominantly emphasized supervised or unsupervised pre-training methods, leaving zero-shot approaches significantly underdeveloped. Existing zero-shot techniques commonly utilize the Segment Anything Model (SAM) in automatic mode or rely on vision-language models to generate cues for segmentation; however, their performances remain unsatisfactory, likely due to the similarity of the camouflaged object and the background. Optical flow, commonly utilized for detecting moving objects, has demonstrated effectiveness even with camouflaged entities. Our method integrates optical flow, a vision-language model, and SAM 2 into a sequential pipeline. Evaluated on the MoCA-Mask dataset, our approach achieves outstanding performance improvements, significantly outperforming existing zero-shot methods by raising the F-measure ($F_\beta^w$) from 0.296 to 0.628. Remarkably, our approach also surpasses supervised methods, increasing the F-measure from 0.476 to 0.628. Additionally, evaluation on the MoCA-Filter dataset demonstrates an increase in the success rate from 0.628 to 0.697 when compared with FlowSAM, a supervised transfer method. A thorough ablation study further validates the individual contributions of each component. More details can be found on https://github.com/weathon/vcos.
BibTeX
@misc{guo2025zs,
  author = {Wenqi Guo and Shan Du},
  title = {{ZS-VCOS: Zero-Shot Outperforms Supervised Video Camouflaged Object Segmentation}},
  year = {2025},
  eprint = {2505.01431},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.01431},
}
Figure