Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement

Yongqing Liang, Xin Li, Navid H. Jafari, Jim ChenNeurIPS 2020

Overview

This paper introduces a robust framework for semi-supervised Video Object Segmentation (VOS) designed to solve two major hurdles: memory bloat in long videos and blurry object boundaries. By implementing an Adaptive Feature Bank (AFB) that dynamically merges and evicts features, alongside an Uncertain-Region Refinement (URR) module for fine-grained boundary correction, the authors achieve state-of-the-art results while maintaining high computational efficiency.

The Challenge

Semi-supervised VOS requires a model to track and mask specific objects across a video, given only the annotation for the first frame. Currently, "matching-based" methods are the gold standard. They store features from previous frames in a "memory bank" and compare them to the current frame to find the target object.

However, these methods face two significant challenges:

  1. The Memory Explosion: As a video progresses, the feature bank grows continuously. Most models either crash due to out-of-memory (OOM) errors on long clips or rely on "static" sampling (e.g., saving every 5th frame), which often misses critical visual changes.
  2. The "Blurry Boundary" Problem: Most VOS models use an encoder-decoder structure. When the mask is upscaled from low-resolution internal features to the original image size, the edges of the object become vague and prone to misclassification.

Proposed Methodology

To address these issues, the authors propose a two-part solution integrated into a matching-based pipeline.

Overall architecture of the proposed VOS framework featuring the Adaptive Feature Bank and Uncertain-Region Refinement modules.

1. Adaptive Feature Bank (AFB)

Instead of simply piling up new features, the AFB manages memory like a smart cache to prevent memory explosion while retaining crucial information:

  • Absorbing: When a new feature is extracted, the model checks if it is similar to an existing one in the bank. If it is (which happens about 90% of the time), it performs a weighted average to update the existing feature rather than adding a redundant one.
  • Discarding: To keep the bank within a fixed budget, the model uses a Least Frequently Used (LFU) index. Features that aren't actively helping the matching process are evicted, allowing the model to handle videos of arbitrary length without OOM errors.
Comparison of feature update strategies: traditional methods simply add new features, while the proposed method dynamically merges them to save memory.

2. Uncertain-Region Refinement (URR)

To fix messy boundaries, the authors introduce a "detect and repair" approach:

  • Uncertainty Detection: The model calculates a pixel-wise uncertainty map based on the ratio between the most likely and second-most likely object classes. If the model is "unsure," the pixel is marked for refinement.
  • Local Refinement: For these uncertain pixels, the model examines a small neighborhood of high-confidence pixels. It uses a local similarity mechanism to "pull" the correct classification from nearby certain regions to the uncertain ones, resulting in much crisper edges.

Datasets and Training

The model was trained and evaluated on several benchmarks:

  • DAVIS17: A standard benchmark for multi-object VOS.
  • YouTube-VOS18: A large-scale dataset featuring thousands of videos and many "unseen" object categories.
  • Long Videos: The authors curated a set of three internet videos longer than 1,500 frames to specifically test the Adaptive Feature Bank's endurance.
  • Pre-training: The model was initially trained on synthetic videos generated from static image datasets (like COCO and Pascal VOC).

Key Insights

  • Redundancy is the Enemy: The finding that 90% of new features can be merged into existing ones without losing accuracy is a significant breakthrough. It proves that VOS memory doesn't need to be infinite; it just needs to be representative.
  • Dynamic Beats Static: While previous state-of-the-art models (like STM) use fixed intervals for memory, this paper shows that a dynamic, LFU-based approach is far more robust for real-world, long-form content where objects might disappear and reappear.
  • Uncertainty as a Guide: Instead of trying to make the whole network higher resolution—which is computationally expensive—the URR module only spends extra effort on the pixels where the model is actually struggling.

Results

  • State-of-the-Art Performance: On DAVIS17, the model achieved a $J\&F$ mean score of 74.6%, outperforming the previous leader (STM) at 71.6%.
  • Long Video Dominance: On the 1,500+ frame test set, the model maintained a $J\&F$ of 83.3%, while competing models dropped significantly in accuracy or failed to run entirely due to memory constraints.
  • Efficiency: Despite the added refinement steps, the model runs at 4.0 FPS on an NVIDIA 1080Ti, which is faster than the 3.4 FPS of the baseline STM model.
  • Generalization: The model performed particularly well on "unseen" objects in the YouTube-VOS dataset, suggesting the matching mechanism learns general appearance cues rather than just memorizing specific categories.

Limitations and Future Work

  • Temporal Smoothness: The model does not explicitly use temporal smoothness assumptions (like motion vectors). While this makes training easier, adding motion cues could potentially reduce "jitter" in very fast-moving sequences.
  • Memory Budget Tuning: While the AFB allows for a fixed memory budget, the paper doesn't deeply explore the trade-off curve between extremely small budgets and accuracy.
  • First-Frame Dependency: Like all semi-supervised VOS approaches, the model is heavily dependent on the quality of the first-frame mask. If the initial annotation is poor, errors will propagate throughout the video.

Summary updated

BibTeX
@inproceedings{liang2020video,
  author = {Yongqing Liang and Xin Li and Navid H. Jafari and Jim Chen},
  title = {{Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure