Background Subtraction With Real-Time Semantic Segmentation

Dongdong Zeng, Xiang Chen, Ming Zhu, Michael Goesele, Arjan KuijperIEEE Access 2019 · DOI

TL;DR

This paper introduces a novel background subtraction framework (RTSS) that combines a traditional background subtraction (BGS) algorithm with a real-time semantic segmentation (RT-SS) module. By using semantic information to refine BGS outputs and feed them back into the background model update process, RTSS achieves state-of-the-art performance among unsupervised methods and even outperforms some supervised deep learning approaches, all while operating in real-time.

What Problem Does This Paper Address?

Accurate and fast foreground object extraction is crucial for many computer vision tasks like surveillance and tracking. Traditional background subtraction (BGS) methods struggle with challenging real-world scenarios such as illumination changes, dynamic backgrounds, and shadows, often leading to false positives and negatives. While semantic segmentation has shown great robustness to these issues, it's typically computationally expensive and has been used as a post-processing step. This paper aims to bridge this gap by proposing a framework that integrates real-time semantic segmentation with BGS, leveraging the strengths of both to improve foreground detection accuracy and robustness in real-time.

Methods

The core of the proposed method, RTSS, is a parallel, two-threaded framework:

  1. BGS Segmenter (B): This component performs traditional background subtraction to generate a preliminary foreground/background (FG/BG) mask. The authors use SuBSENSE as a benchmark BGS algorithm due to its state-of-the-art performance among unsupervised methods.
  2. Semantic Segmenter (S): This component extracts object-level semantic information from the input frames. To achieve real-time performance, they suggest two strategies: using inherently real-time semantic segmentation algorithms (like ICNet, ERFNet, LinkNet) or performing segmentation every N frames. ICNet is used as the primary benchmark semantic segmenter.

The key innovation lies in the feedback mechanism: the refined FG/BG mask (Dt), generated by combining the outputs of B and S, is fed back to update the background model. This allows the semantic information to actively guide and improve the background model's adaptation and accuracy over time, addressing issues that traditional BGS methods alone cannot.

Datasets

  • Training and Evaluation: The primary dataset used is the Change Detection (CDnet) 2014 dataset. This dataset is described as a region-level benchmark with realistic scenarios and accurate ground truths, comprising 53 video sequences across 11 categories (e.g., baseline, dynamic background, shadow).
  • Semantic Segmentation Training: The semantic segmentation model (ICNet) was trained on the ADE20K dataset, a large-scale dataset for scene parsing with 150 object classes.

Key Insights and Ideas

  • Synergy of BGS and Semantic Segmentation: The paper demonstrates that combining the robustness of semantic segmentation with the real-time adaptability of BGS leads to significant improvements over using either method alone.
  • Feedback Loop is Crucial: Simply using semantic segmentation as a post-processing step (as in prior work) is less effective than integrating it into the BGS model update process. The feedback loop allows semantic context to directly refine the background model.
  • Real-time Semantic Segmentation is Key: The ability to perform semantic segmentation in real-time (or with strategic subsampling) is essential for the proposed framework to achieve its performance goals without prohibitive computational cost.
  • Flexibility of the Framework: The modular design allows for easy replacement of both the BGS and semantic segmentation components, making the framework adaptable to future advancements in either field.

Key Results

  • State-of-the-Art Unsupervised Performance: The proposed RTSS framework (using SuBSENSE + ICNet) achieved state-of-the-art performance among unsupervised BGS methods on the CDnet 2014 dataset, with an overall F-Measure of 0.7649, a 3% improvement over the baseline SuBSENSE.
  • Outperforms Supervised Methods: In some categories, RTSS even outperformed certain deep learning-based supervised algorithms.
  • Significant Improvements in Challenging Categories: Notable improvements were seen in "dynamic background" (11% F-Measure increase), "intermittent object motion" (36% increase), and "shadow" categories.
  • PSPNet Integration: Using a more powerful semantic segmentation model like PSPNet (even with frame subsampling) further boosted performance, achieving an overall F-Measure of 0.8030 (8% improvement over baseline).
  • Real-time Operation: The RTSS framework with ICNet achieved an average processing time of 38 ms per frame, enabling real-time performance.

Limitations and Future Work

  • Semantic Segmentation Accuracy: The paper notes that when the semantic segmentation accuracy is not high (e.g., in the "turbulence" category), the performance improvement of RTSS can be limited. This suggests that the quality of the semantic segmentation directly impacts the overall system's effectiveness.
  • Computational Cost of Advanced Models: While PSPNet offers better accuracy, its higher computational cost necessitates frame subsampling (N>1) to maintain near real-time performance.
  • Future Work: The authors suggest exploring more efficient fusion strategies between BGS and semantic segmentation outputs, as well as integrating faster and more accurate BGS and semantic segmentation algorithms as they emerge. They also mention the potential for further improvements by developing more precise semantic segmentation algorithms.

Summary updated

Abstract

Accurate and fast foreground (FG) object extraction is very important for object tracking and recognition in video surveillance. Although many background subtraction (BGS) methods have been proposed in the recent past, it is still regarded as a tough problem due to the variety of challenging situations that occur in real-world scenarios. In this paper, we explore this problem from a new perspective and propose a novel BGS framework with the real-time semantic segmentation. Our proposed framework consists of two components, a traditional BGS segmenter B and a real-time semantic segmenter S. The BGS segmenter B aims to construct background (BG) models and segments FG objects. The real-time semantic segmenter S is used to refine the FG segmentation outputs as feedbacks for improving the model updating accuracy. B and S work in parallel on two threads. For each input frame It, the BGS segmenter B computes a preliminary FG/BG mask B <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> . At the same time, the real-time semantic segmenter S extracts the object-level semantics S <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> . Then, some specific rules are applied on B <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> and S <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> to generate the final detection D <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> . Finally, the refined FG/BG mask D <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> is fed back to update the BG model. The comprehensive experiments evaluated on the CDnet 2014 dataset demonstrate that our proposed method achieves the state-of-the-art performance among all unsupervised BGS methods while operating at the real-time and even performs better than some deep learning-based supervised algorithms. In addition, our proposed framework is very flexible and has the potential for generalization.
BibTeX
@inproceedings{zeng2019background,
  author = {Dongdong Zeng and Xiang Chen and Ming Zhu and Michael Goesele and Arjan Kuijper},
  title = {{Background Subtraction With Real-Time Semantic Segmentation}},
  booktitle = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2899348},
}
Figure