Revisiting Click-Based Interactive Video Object Segmentation

Stéphane Vujasinovic, Sebastian Bullinger, Stefan Becker, Norbert Scherer-Negenborn, Michael Arens, Rainer Stiefelhagen

TL;DR

The CiVOS (Click-based interactive Video Object Segmentation) framework shifts the paradigm of interactive video editing from tedious "scribbling" to simple "clicking." By decoupling the interaction and propagation tasks and introducing a hardware-independent evaluation metric, the authors demonstrate that clicks can achieve competitive accuracy with significantly less user effort.

What Problem Does This Paper Address?

Interactive Video Object Segmentation (iVOS) is the process where a human helps a computer identify and track an object through a video. Traditionally, this involves drawing "scribbles" over the object. While effective, scribbling is slow—taking about 11 seconds per object—and tedious for large datasets.

Furthermore, the standard way we measure these systems (Area Under the Curve or AUC) is "hardware-dependent." Because the metric factors in the time taken to process frames, a researcher with a $10,000 GPU will report better scores than one with a mid-range card, even if their algorithm is identical. This makes it nearly impossible to compare different methods fairly across different research labs.

Methods

The authors propose CiVOS, a modular framework consisting of two main engines:

  1. Interaction Module: This uses a state-of-the-art image segmentation model (RITM) with an HRNet backbone. It takes a user's click (positive for the object, negative for the background) and the current frame to generate a precise object mask.
  2. Propagation Module: Based on the MiVOS architecture, this module uses a "Space-Time-Memory" approach. It takes the mask from the interaction frame and "spreads" it forward and backward through the video, using a memory bank of previous frames to maintain consistency.
  3. Difference-Aware Fusion: To ensure that new user corrections don't get lost when merged with previous results, a learnable fusion block is used to integrate the new interaction mask with the existing video sequence.

To bridge the gap between existing scribble-based datasets and their click-based approach, the authors developed three strategies ($f_1, f_2, f_3$) to simulate human clicks by calculating the centers of scribbles or the centers of incorrectly segmented regions.

Datasets

The study primarily uses the DAVIS 2017 interactive benchmark, which is the gold standard for this field. Since DAVIS provides scribble-based annotations, the authors adapted it for click-based evaluation using their simulation strategies. They have made their adjusted evaluation framework and source code publicly available to encourage reproducible research.

Key Insights and Ideas

  • Efficiency is King: A click takes 1–3 seconds, while a scribble takes ~11 seconds. By proving that clicks can match scribble performance, the authors show a path toward reducing human annotation workload by 70-90%.
  • The "R-AUC" Metric: The authors propose R-AUC-$\mathcal{J}\&\mathcal{F}$, which measures accuracy based on the number of interaction rounds rather than clock time. This removes hardware bias, allowing for "hardware-independent" comparisons between algorithms.
  • Diminishing Returns of Interaction: Their evaluation shows that after about three clicks per object per round, accuracy gains plateau. This suggests a "sweet spot" for UI design: don't ask the user for too much detail in one go.
  • Error-Centric Clicking: The most effective strategy ($f_3$) was placing clicks in the center of "erroneous regions" (areas the computer got wrong). This is more intuitive and effective than simply clicking the center of the object.

Key Results

  • Competitive Performance: CiVOS achieved an R-AUC of 0.76 and a standard AUC of 0.83 on the DAVIS 2017 validation set.
  • Better than Click-based Baselines: When compared to a version of the popular MiVOS model modified to accept clicks, CiVOS outperformed it significantly (0.76 vs 0.70 R-AUC).
  • Closing the Gap: While scribble-based MiVOS still holds a slight edge in raw accuracy (0.81 R-AUC), CiVOS reaches a very similar level of quality with a much lower "cost" of user time.

Limitations and Future Work

The authors acknowledge that while their "robot agent" simulates clicks well, it doesn't perfectly capture the unpredictability of a human user. Future work could involve user studies to see how humans interact with the system in real-time.

Additionally, the current propagation process is somewhat independent of the masks inferred in previous rounds. Improving the "temporal memory" so the system remembers why a user clicked a certain spot several rounds ago could lead to even more robust tracking in complex scenes with heavy occlusion.

Summary updated

Abstract

While current methods for interactive Video Object Segmentation (iVOS) rely on scribble-based interactions to generate precise object masks, we propose a Click-based interactive Video Object Segmentation (CiVOS) framework to simplify the required user workload as much as possible. CiVOS builds on de-coupled modules reflecting user interaction and mask propagation. The interaction module converts click-based interactions into an object mask, which is then inferred to the remaining frames by the propagation module. Additional user interactions allow for a refinement of the object mask. The approach is extensively evaluated on the popular interactive~DAVIS dataset, but with an inevitable adaptation of scribble-based interactions with click-based counterparts. We consider several strategies for generating clicks during our evaluation to reflect various user inputs and adjust the DAVIS performance metric to perform a hardware-independent comparison. The presented CiVOS pipeline achieves competitive results, although requiring a lower user workload.
BibTeX
@inproceedings{vujasinovic2022revisiting,
  author = {Stéphane Vujasinovic and Sebastian Bullinger and Stefan Becker and Norbert Scherer-Negenborn and Michael Arens and Rainer Stiefelhagen},
  title = {{Revisiting Click-Based Interactive Video Object Segmentation}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897460},
}

Loading related content...

Figure