Abstract
Video segmentation is the foundation of many senior computer vision applications. For robotic vision applications, video object segmentation is facing more difficulties. Since the videos are usually collected from those complicated scenes and the camera perspective may be static for some time, object saliency and motion clues which are widely used in traditional video segmentation approaches could be unreliable. Besides, its online processing ability is of crucial importance. In this paper, we propose an online video object segmentation method, which is organized as a segmentation-by-detection framework. It first effectively segments out meaningful objects in short video frame batches with the help of object detectors, and then the segmentation of different batches can be associated with each other via bi-directional notebook based connecting strategy. Experiments conducted on public available datasets verify the above-mentioned property and show good performance of the proposed method.
Introduction
Traditional video object segmentation (VOS) often relies on two primary assumptions: that either the camera or the object is moving to provide motion cues, and that a single "primary" object is consistently present. However, robotic vision frequently breaks these assumptions. A service robot might remain stationary while observing a scene, rendering motion-based algorithms ineffective. Furthermore, robots require "online" processing—the ability to segment video as it streams—rather than offline optimization over an entire sequence.
To address these challenges, this research introduces an online "segmentation-by-detection" framework. By using semantic object detectors to anchor the process and a novel "bi-directional notebook" to link objects across video batches, the system can reliably segment multiple objects even in complex, static indoor environments.
The Segmentation-by-Detection Framework
The proposed framework moves from coarse detections to fine-grained pixel masks through a multi-stage pipeline designed for streaming data.
1. Overlapping Batch Partitioning
Instead of processing single frames (which lack context) or the entire video (which isn't online), the system processes small, overlapping batches. This provides necessary temporal context while maintaining a streaming workflow.
2. Tracklet-Based Verification
The system utilizes the YOLO detector to identify objects. To filter out transient "ghost" detections or flickers, it constructs short-term tracklets. Detections that do not persist across the batch are discarded, while missing detections in intermediate frames are "filled in" using tracking logic.
3. Co-Segmentation and Refinement
Within each batch, the system selects the best object proposals (using the LPO algorithm) that align with the verified detection boxes. A shortest-path algorithm ensures these proposals remain consistent across frames. For final pixel-level accuracy, a modified GrabCut refinement incorporates spatial-temporal constraints, such as optical flow, to ensure sharp and consistent mask boundaries.
4. Notebook-Based Bi-Directional Connection (NBDC)
To maintain object identity across different batches, the system maintains a "notebook" (a memory bank) of previously observed objects. It employs a bi-directional matching strategy based on color histograms and location. A new segment is linked to an existing "word" in the notebook only if both the segment and the notebook entry identify each other as the best match, significantly reducing identity switches.
Experimental Results
The method was evaluated on two benchmarks: Safari_1.0 (outdoor videos with multiple targets) and RGB-D Scenes Dataset v.2 (challenging indoor scenes with household objects like bowls and mugs).
Key findings include:
- Superior Discovery: On the RGB-D Scenes dataset, the method discovered significantly more object instances (NDI) than traditional methods like SGBH or PGP, even when those methods were augmented with YOLO detections.
- Accuracy and Consistency: The framework achieved a higher Mean IoU on Safari_1.0 compared to state-of-the-art offline methods. The NBDC strategy proved highly effective at tracking objects through appearances and disappearances, with the number of connection clips closely matching ground truth.
- Robustness to Stasis: By anchoring the process in semantic detection rather than motion saliency, the system remains effective even when both the robot and the target objects are stationary.
Discussion and Future Work
While the framework is robust, it currently depends heavily on the initial detector's performance; if YOLO fails due to extreme lighting or blur, the pipeline cannot initiate. Additionally, while the logic is "online," the current processing speed of approximately 8.25 seconds per frame is not yet real-time for high-speed maneuvers. Future work will focus on improving computational efficiency and incorporating more robust tracking to handle long-term occlusions.
Summary updated