MBS-NET: A MOVING-CAMERA BACKGROUND SUBTRACTION NETWORK FOR AUTONOMOUS DRIVING

Jiayu Wei, Jing Jiang, Alper Yılmaz2021 · DOI

Abstract

Background subtraction is a fundamental computer vision task that identifies moving objects (foreground) by isolating them from the static environment (background). While traditional methods rely on stationary cameras to learn a semi-constant background, autonomous driving presents a significant challenge: the camera itself is in motion. This paper introduces MBS-Net (Moving-camera Background Subtraction Network), the first deep learning approach specifically designed for background subtraction in moving-camera scenarios. By leveraging temporal cues through Conditional Random Fields (CRFs) and optimizing for extreme class imbalances with Focal Loss, MBS-Net achieves high-accuracy foreground detection without requiring pre-training on specific object categories.

Introduction: The Moving Camera Challenge

In traditional surveillance, background subtraction is straightforward because the background remains largely static. However, for a vehicle-embedded camera, every pixel changes as the car moves. Most modern autonomous systems attempt to solve this by using object detection to identify known categories like "car" or "pedestrian."

Example of background subtraction on the ApolloScape dataset. The top image shows the original frame, and the bottom shows the resulting foreground mask.

This "object-centric" approach has a major flaw: it can only detect what it has been trained to see. If an unusual obstacle—a fallen tree or a stray animal—appears, the system may fail. MBS-Net adopts a "background-centric" philosophy. Since the variety of background elements (roads, sky, buildings) is more predictable than the infinite variety of potential road obstacles, modeling the background provides a more robust safety net for autonomous navigation.

The MBS-Net Architecture

MBS-Net is built upon the Bilateral Segmentation Network (BiSeNet) architecture, which utilizes two distinct paths to balance detail and context. The Spatial Path preserves high-resolution spatial information, while the Context Path uses downsampling to capture a broader field of view.

The MBS-Net architecture, featuring Attention Refinement (ARM) and Feature Fusion (FFM) modules to integrate spatial and contextual information.

To adapt this architecture for background subtraction, the authors introduced three key innovations:

  1. Temporal Regularization via CRFs: Unlike standard spatial CRFs that sharpen edges within a single frame, MBS-Net integrates a specialized CRF layer that models interactions between the current frame and previous frames. This ensures temporal consistency, effectively filtering out noise caused by camera vibrations or sudden lighting shifts.
  2. Attention and Fusion Modules: The network employs Attention Refinement Modules (ARM) to refine features in the context path and a Feature Fusion Module (FFM) to combine the outputs of the spatial and context paths efficiently.
  3. Focal Loss for Class Imbalance: In driving scenes, background pixels (road, sky) vastly outnumber foreground pixels. Standard loss functions often ignore the small foreground objects. Focal Loss applies dynamic weights, forcing the network to prioritize these "hard" foreground examples.

Experimental Results

The researchers evaluated MBS-Net using the ApolloScape dataset, utilizing over 11,000 high-resolution frames (3384 × 2710). The results demonstrate the power of the temporal CRF integration:

  • Accuracy: Using an Xception39 backbone, the model achieved a 97.53% Mean IoU (Intersection over Union) for background and 76.06% for foreground.
  • The CRF Impact: Activating the CRF layer resulted in a significant performance boost, raising foreground detection accuracy from 68.35% to 76.06%.
  • Real-time Potential: Without the CRF layer, the ResNet18-based version of the network is incredibly fast, reaching 305 FPS. However, the computationally intensive CRF layer drops the speed to approximately 3.3 - 3.6 FPS, highlighting a trade-off between temporal accuracy and processing speed.

Conclusion and Future Work

MBS-Net proves that modeling the background is a viable and efficient alternative to traditional object detection for autonomous vehicles. By treating background subtraction as a sequence problem rather than a frame-by-frame task, the network achieves superior robustness against camera motion.

Future research will focus on overcoming the computational bottleneck of the CRF layer, potentially using reinforcement learning or more efficient regularization techniques to maintain high accuracy at real-time speeds. Additionally, the authors plan to expand the system into "semantic" background subtraction, where the background is categorized into specific types like "sidewalk" versus "drivable road."

Summary updated

Abstract

Background subtraction aims at detecting salient background which in return provides regions of moving objects referred to as the foreground. Background subtraction inherently uses the temporal relations by including time dimension in its formulation. Alternative techniques to background subtraction require stationary cameras for learning the background. Stationary cameras provide semi-constant background images that make learning salient background easier. Still cameras, however, are not applicable to moving camera scenarios, such as vehicle embedded camera for autonomous driving. For moving cameras, due to the complexity of modelling changing background, recent approaches focus on directly detecting the foreground objects in each frame independently. This treatment, however, requires learning all possible objects that can appear in the field of view. In this paper, we achieve background subtraction for moving cameras using specialized deep learning approach, the Moving-camera Background Subtraction Network (MBS-Net). Our approach is robust to detect changing background in various scenarios and does not require training on foreground objects. The developed approach uses temporal cues from past frames by applying Conditional Random Fields as a part of the developed neural network. Our proposed method have a good performance on ApolloScape dataset (Huang et al., 2018) with resolution 3384 × 2710 videos. To the best of our acknowledge, this paper is the first to propose background subtraction for moving cameras using deep learning.
BibTeX
@inproceedings{wei2021mbs,
  author = {Jiayu Wei and Jing Jiang and Alper Yılmaz},
  title = {{MBS-NET: A MOVING-CAMERA BACKGROUND SUBTRACTION NETWORK FOR AUTONOMOUS DRIVING}},
  booktitle = {˜The},
  year = {2021},
  doi = {10.5194/isprs-archives-xliii-b2-2021-69-2021},
}
Figure