Revisiting Click-based Interactive Video Object Segmentation

Stéphane Vujasinovic, Sebastian Bullinger, Stefan Becker, Norbert Scherer-Negenborn, Michael Arens, Rainer Stiefelhagen

Abstract

While current methods for interactive Video Object Segmentation (iVOS) rely on scribble-based interactions to generate precise object masks, we propose a Click-based interactive Video Object Segmentation (CiVOS) framework to simplify the required user workload as much as possible. CiVOS builds on de-coupled modules reflecting user interaction and mask propagation. The interaction module converts click-based interactions into an object mask, which is then inferred to the remaining frames by the propagation module. Additional user interactions allow for a refinement of the object mask. The approach is extensively evaluated on the popular interactive DAVIS dataset, but with an inevitable adaptation of scribble-based interactions with click-based counterparts. We consider several strategies for generating clicks during our evaluation to reflect various user inputs and adjust the DAVIS performance metric to perform a hardware-independent comparison. The presented CiVOS pipeline achieves competitive results, although requiring a lower user workload.

The Problem: Scribbles and Hardware Bias

Interactive Video Object Segmentation (iVOS) is a collaborative process where human input helps a system identify and track objects across video frames. Traditionally, this input takes the form of "scribbles." While effective, scribbling is time-intensive—averaging 11 seconds per object—making it tedious for large-scale datasets.

Furthermore, standard evaluation metrics like Area Under the Curve (AUC) are often hardware-dependent. Because these metrics factor in processing time, results can be skewed by the power of the researcher's GPU rather than the efficiency of the algorithm itself. This makes fair comparison across different research environments nearly impossible.

The CiVOS Framework

The proposed CiVOS framework addresses these issues by shifting to a click-based interaction model and decoupling the segmentation process into two distinct engines:

  1. Interaction Module: Utilizing a state-of-the-art image segmentation model (RITM) with an HRNet backbone, this module converts simple user clicks (positive for the object, negative for the background) into precise object masks.
  2. Propagation Module: Based on a Space-Time-Memory architecture, this module "spreads" the mask from the interaction frame to the rest of the video sequence, using a memory bank of previous frames to maintain temporal consistency.
  3. Difference-Aware Fusion: A learnable fusion block integrates new user corrections with existing results, ensuring that refinements are accurately merged into the video sequence without losing previous progress.
The CiVOS architecture featuring decoupled Interaction and Propagation modules with Difference-Aware Fusion.

Hardware-Independent Evaluation

To enable fair comparisons, the authors introduce the R-AUC-$\mathcal{J}\&\mathcal{F}$ metric. Unlike standard AUC, R-AUC measures accuracy based on the number of interaction rounds rather than clock time. This removes hardware bias from the equation.

To bridge the gap with existing scribble-based benchmarks like DAVIS 2017, the authors developed three simulation strategies ($f_1, f_2, f_3$) to mimic human clicking behavior. The most effective strategy, $f_3$, involves placing clicks in the center of erroneous regions, which proved more effective than simply clicking the center of the object.

Key Results and Insights

CiVOS demonstrates that clicks can achieve competitive accuracy with significantly less user effort. Key findings include:

  • Efficiency: A click takes only 1–3 seconds compared to 11 seconds for a scribble, representing a potential 70-90% reduction in human workload.
  • Competitive Performance: CiVOS achieved an R-AUC of 0.76, significantly outperforming click-based adaptations of previous state-of-the-art models like MiVOS.
  • Diminishing Returns: Accuracy gains typically plateau after approximately three clicks per object per round, suggesting an optimal "sweet spot" for user interaction design.
Quantitative comparison of CiVOS against state-of-the-art scribble and click-based methods on DAVIS 2017.

Conclusion

CiVOS proves that click-based interaction is a viable, more efficient alternative to scribbling for interactive video segmentation. By decoupling interaction from propagation and introducing the hardware-independent R-AUC metric, this work provides a more accessible and fair framework for future iVOS research. Future work will focus on improving "temporal memory" to better handle complex scenes with heavy occlusion and conducting real-world user studies.

Summary updated

Abstract

While current methods for interactive Video Object Segmentation (iVOS) rely on scribble-based interactions to generate precise object masks, we propose a Click-based interactive Video Object Segmentation (CiVOS) framework to simplify the required user workload as much as possible. CiVOS builds on de-coupled modules reflecting user interaction and mask propagation. The interaction module converts click-based interactions into an object mask, which is then inferred to the remaining frames by the propagation module. Additional user interactions allow for a refinement of the object mask. The approach is extensively evaluated on the popular interactive~DAVIS dataset, but with an inevitable adaptation of scribble-based interactions with click-based counterparts. We consider several strategies for generating clicks during our evaluation to reflect various user inputs and adjust the DAVIS performance metric to perform a hardware-independent comparison. The presented CiVOS pipeline achieves competitive results, although requiring a lower user workload.
BibTeX
@inproceedings{vujasinovic2022revisiting,
  author = {Stéphane Vujasinovic and Sebastian Bullinger and Stefan Becker and Norbert Scherer-Negenborn and Michael Arens and Rainer Stiefelhagen},
  title = {{Revisiting Click-based Interactive Video Object Segmentation}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897460},
}
Figure