Structure Matters: Revisiting Boundary Refinement in Video Object Segmentation

Guanyi Qin, Ziyue Wang, Daiyun Shen, Haofeng Liu, Hantao Zhou, Junde Wu, Runze Hu, Yueming Jin

Abstract

Semi-supervised Video Object Segmentation (SVOS) has advanced significantly through memory-based feature matching. However, current state-of-the-art models often exhibit a "blind spot" regarding geometric structure, leading to failures during occlusions or in scenes with low foreground-background contrast. We introduce OASIS (bOundary Amendment video object Segmentation with Inherent Structure refinement), a framework designed to explicitly model object boundaries. By fusing traditional Canny edge priors with deep memory features and utilizing evidential learning for uncertainty estimation, OASIS achieves a 91.6 J&F score on DAVIS-17 while operating at 48 FPS.

Introduction

Semi-supervised Video Object Segmentation (SVOS) is a fundamental task in computer vision, often described as "tracking by segmenting." Given an initial mask for an object, the system must track and segment that object throughout a video sequence. While recent memory-based models have set new benchmarks by matching current frame features to a historical "memory bank," they frequently struggle with boundary precision. This is particularly evident when objects overlap or when foreground and background appearances are highly similar.

The Structural Gap in SVOS

The primary limitation of existing SVOS methods lies in their heavy reliance on appearance consistency at the expense of inherent geometric structure. When an object is partially occluded, its boundaries become fragmented in the feature space. Without a robust mechanism to maintain "object-ness" or structural integrity, models fail to distinguish where one object ends and another begins. This paper argues that revisiting traditional structural priors can bridge this gap.

Proposed Methodology: The OASIS Framework

The OASIS framework introduces two core innovations to the standard VOS pipeline: the Inherent Structure Refinement (ISR) module and Evidential Learning (EDL) for uncertainty quantification.

The OASIS Framework Overview, integrating Structure Refinement and Evidential Learning.

Inherent Structure Refinement (ISR)

The ISR module aims to generate a "skeleton" of the target object to guide the segmentation process. It begins by applying a classic Canny edge detector to the input image to extract raw edge priors. Recognizing that global edges contain significant background noise, the framework employs a lightweight Structure Decoder. This decoder fuses the raw edges with object-specific features retrieved from the memory bank. The resulting target-level structure map highlights the boundaries of the specific object being tracked, which is then used to re-weight image features and sharpen the final segmentation mask.

Comparison of structure decoding with and without memory feature fusion.

Evidential Learning for Uncertainty

To handle ambiguous regions, such as overlapping boundaries, OASIS incorporates Evidential Learning (EDL). Unlike standard models that output "hard" probabilities, EDL treats the model's output as a Dirichlet distribution. This allows the system to quantify uncertainty at the pixel level. By introducing a specific loss term to minimize this uncertainty, the model becomes more robust in "confusing" scenarios, preventing the "bleeding" of masks across object boundaries.

Experimental Evaluation

OASIS was rigorously tested across four major benchmarks: DAVIS-2017, YouTubeVOS 2019, and the challenging MOSE dataset.

  • Accuracy: OASIS achieved a state-of-the-art 91.6 J&F on DAVIS-17 and 86.6 G on YouTubeVOS 2019, outperforming previous leaders like the Cutie model.
  • Robustness: On the MOSE dataset, which features crowded scenes and frequent occlusions, OASIS reached 62.1 J&F, demonstrating its superior handling of complex structural interactions.
  • Efficiency: Despite the addition of structural refinement, the model remains highly efficient. The structure decoder adds only approximately 2 million parameters, allowing OASIS to maintain a real-time processing speed of 48 FPS.

Discussion and Conclusion

The success of OASIS highlights a crucial insight: traditional computer vision techniques, like Canny edge detection, still hold significant value when integrated thoughtfully into deep learning architectures. By filtering global structural priors through target-specific memory queries, OASIS effectively isolates the "inherent structure" of objects. Furthermore, by "knowing what it doesn't know" through Evidential Learning, the model avoids overconfidence in ambiguous regions. Future work may explore more advanced edge detection methods or alternative uncertainty-aware learning patterns to further refine boundary accuracy in SVOS.

Summary updated

BibTeX
@inproceedings{qin2025structure,
  author = {Guanyi Qin and Ziyue Wang and Daiyun Shen and Haofeng Liu and Hantao Zhou and Junde Wu and Runze Hu and Yueming Jin},
  title = {{Structure Matters: Revisiting Boundary Refinement in Video Object Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure