Semantic Instance Meets Salient Object: Study on Video Semantic Salient Instance Segmentation

Trung-Nghia Le, Akihiro SugimotoarXiv 2018 · arXiv

TL;DR

This paper introduces a new computer vision task: Video Semantic Salient Instance Segmentation (VSSIS), which aims to simultaneously identify, track, and categorically label only the most prominent (salient) objects in a video. To tackle this, the authors propose SISO (Semantic Instance - Salient Object), a novel two-stream framework that fuses semantic instance segmentation with salient object detection, alongside a newly curated benchmark dataset named SESIV.

What Problem Does This Paper Address?

In applications like autonomous driving and robotic navigation, processing every single object in a visual scene is computationally wasteful. Instead, systems benefit from focusing exclusively on salient objects—the prominent foreground actors like a pedestrian stepping onto the street or a vehicle passing by.

However, existing techniques fall short of this goal:

  • Salient Object Segmentation (SOS) merely identifies a region as "salient" or "non-salient" without distinguishing between multiple interacting objects or identifying what they are.
  • Semantic Instance Segmentation (SIS) detects and classifies all individual objects but treats background clutter and critical foreground actors with equal weight.

To bridge this gap, the authors define the VSSIS task, which requires a system to segment individual prominent objects, track them consistently across video frames (even through occlusions), and assign them a semantic class label.

Figure 1: Segmentation levels of salient objects. The input video frame is followed by different levels of label anno- tation. Our work focuses on segmenting semantic salient instances (most right).

Methods

The authors introduce the SISO (Semantic Instance - Salient Object) framework, a simple yet effective two-stream pipeline that leverages the strengths of both SIS and SOS methods.

Figure 1: Segmentation levels of salient objects. The input video frame is followed by different levels of label anno- tation. Our work focuses on segmenting semantic salient instances (most right).

The framework consists of three core components:

  1. Sequential Fusion: Standard instance segmentation methods often struggle with overlapping object proposals. SISO introduces a deterministic merging order based on how well semantic instance proposals overlap with the binary mask generated by the salient object stream (using Intersection over Union, or IoU). This allows the system to cleanly decompose the salient region into non-overlapping instances without requiring first-frame ground-truth labels.
  2. Recurrent Instance Propagation: Video frames often suffer from motion blur or severe object deformation, causing standard frame-by-frame segmenters to miss objects. SISO uses optical flow (via FlowNet2) to recurrently propagate high-confidence object masks from clear frames to adjacent lower-confidence frames, running iteratively until the video's overall confidence score converges.
  3. Identity Tracking: To maintain long-term consistency, SISO combines short-term Identity Propagation (warping object identities to the next frame via optical flow) with a Re-identification module. If an object is occluded or leaves the frame, an instance search mechanism uses a Faster R-CNN feature extractor and cosine similarity to re-identify the object when it reappears. Finally, a Semantic Unification step aggregates classification scores across the entire video to ensure an object's category label remains uniform over time.

Datasets

Because no dataset existed for this specific task, the authors introduced the SEmantic Salient Instance Video (SESIV) dataset.

Built on top of the popular DAVIS-2017 video segmentation benchmark, the authors manually filtered out non-salient objects and mapped the remaining prominent instances to 29 categories from the MS-COCO dataset (such as person, dog, car, horse). The resulting dataset contains 84 high-quality video sequences with pixel-accurate, per-frame ground-truth labels for saliency, instance IDs, and semantic categories.

Figure 3: Samples from the SESIV dataset. From top to bottom, video frame is followed by saliency ground-truth, instance ground-truth, and semantic label ground-truth.

Key Insights and Ideas

  • Saliency as a Natural Filter: By using a salient object detection stream as a mask, the framework effectively filters out the "noise" of background objects that standard instance segmenters pick up, drastically narrowing down the focus to critical actors.
  • Bidirectional Temporal Refinement: The recurrent propagation mechanism demonstrates that temporal context isn't just useful for tracking; it can actively reconstruct missing spatial data in degraded frames by passing mask information both forward and backward through time.
  • Resolving Overlaps Without Supervision: Instead of relying on complex learned topological relationships or initial-frame ground truths to handle overlapping instances, a simple greedy matching strategy based on saliency-mask IoU proves remarkably robust.

Key Results

The framework was evaluated using two primary metrics: semantic region similarity ($JS$) and semantic contour accuracy ($FS$). The authors tested SISO using two different underlying architectures for the SIS stream: Mask R-CNN and Multi-task Network Cascades (MNC).

  • Massive Baseline Improvements: Applying Mask R-CNN frame-by-frame without temporal tracking yielded a dismal $JS$ of 0.41. Integrating the full SISO framework skyrocketed the performance to a $JS$ of 41.71 and an $FS$ of 42.59.
  • Ablation Validations:
    • Adding Recurrent Instance Propagation to the sequential fusion baseline improved the $JS$ score from 37.43 to 41.71.
  • The inclusion of the Re-identification module alongside basic identity propagation raised the $JS$ score from 33.74 to 41.71, proving the necessity of long-term tracking mechanisms to handle occlusions.

Limitations and Future Work

While SISO establishes a strong baseline, it is fundamentally a multi-stage, two-stream pipeline that relies on separate pre-trained networks for saliency, instance segmentation, optical flow, and re-identification. This modular nature prevents end-to-end training and limits real-time efficiency. The authors note that the next logical step for the community is developing unified, single-stage architectures capable of directly segmenting semantic salient instances from video streams.

Summary updated

Abstract

Focusing on only semantic instances that only salient in a scene gains more benefits for robot navigation and self-driving cars than looking at all objects in the whole scene. This paper pushes the envelope on salient regions in a video to decompose them into semantically meaningful components, namely, semantic salient instances. We provide the baseline for the new task of video semantic salient instance segmentation (VSSIS), that is, Semantic Instance - Salient Object (SISO) framework. The SISO framework is simple yet efficient, leveraging advantages of two different segmentation tasks, i.e. semantic instance segmentation and salient object segmentation to eventually fuse them for the final result. In SISO, we introduce a sequential fusion by looking at overlapping pixels between semantic instances and salient regions to have non-overlapping instances one by one. We also introduce a recurrent instance propagation to refine the shapes and semantic meanings of instances, and an identity tracking to maintain both the identity and the semantic meaning of instances over the entire video. Experimental results demonstrated the effectiveness of our SISO baseline, which can handle occlusions in videos. In addition, to tackle the task of VSSIS, we augment the DAVIS-2017 benchmark dataset by assigning semantic ground-truth for salient instance labels, obtaining SEmantic Salient Instance Video (SESIV) dataset. Our SESIV dataset consists of 84 high-quality video sequences with pixel-wisely per-frame ground-truth labels.
BibTeX
@misc{le2018semantic,
  author = {Trung-Nghia Le and Akihiro Sugimoto},
  title = {{Semantic Instance Meets Salient Object: Study on Video Semantic Salient Instance Segmentation}},
  year = {2018},
  eprint = {1807.01452},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.01452},
}
Figure