Abstract
Background subtraction is a fundamental computer vision task that identifies moving objects (foreground) by isolating them from the static environment (background). While traditional methods rely on stationary cameras to learn a semi-constant background, autonomous driving presents a significant challenge: the camera itself is in motion. This paper introduces MBS-Net (Moving-camera Background Subtraction Network), the first deep learning approach specifically designed for background subtraction in moving-camera scenarios. By leveraging temporal cues through Conditional Random Fields (CRFs) and optimizing for extreme class imbalances with Focal Loss, MBS-Net achieves high-accuracy foreground detection without requiring pre-training on specific object categories.
Introduction: The Moving Camera Challenge
In traditional surveillance, background subtraction is straightforward because the background remains largely static. However, for a vehicle-embedded camera, every pixel changes as the car moves. Most modern autonomous systems attempt to solve this by using object detection to identify known categories like "car" or "pedestrian."
This "object-centric" approach has a major flaw: it can only detect what it has been trained to see. If an unusual obstacle—a fallen tree or a stray animal—appears, the system may fail. MBS-Net adopts a "background-centric" philosophy. Since the variety of background elements (roads, sky, buildings) is more predictable than the infinite variety of potential road obstacles, modeling the background provides a more robust safety net for autonomous navigation.
The MBS-Net Architecture
MBS-Net is built upon the Bilateral Segmentation Network (BiSeNet) architecture, which utilizes two distinct paths to balance detail and context. The Spatial Path preserves high-resolution spatial information, while the Context Path uses downsampling to capture a broader field of view.
To adapt this architecture for background subtraction, the authors introduced three key innovations:
- Temporal Regularization via CRFs: Unlike standard spatial CRFs that sharpen edges within a single frame, MBS-Net integrates a specialized CRF layer that models interactions between the current frame and previous frames. This ensures temporal consistency, effectively filtering out noise caused by camera vibrations or sudden lighting shifts.
- Attention and Fusion Modules: The network employs Attention Refinement Modules (ARM) to refine features in the context path and a Feature Fusion Module (FFM) to combine the outputs of the spatial and context paths efficiently.
- Focal Loss for Class Imbalance: In driving scenes, background pixels (road, sky) vastly outnumber foreground pixels. Standard loss functions often ignore the small foreground objects. Focal Loss applies dynamic weights, forcing the network to prioritize these "hard" foreground examples.
Experimental Results
The researchers evaluated MBS-Net using the ApolloScape dataset, utilizing over 11,000 high-resolution frames (3384 × 2710). The results demonstrate the power of the temporal CRF integration:
- Accuracy: Using an Xception39 backbone, the model achieved a 97.53% Mean IoU (Intersection over Union) for background and 76.06% for foreground.
- The CRF Impact: Activating the CRF layer resulted in a significant performance boost, raising foreground detection accuracy from 68.35% to 76.06%.
- Real-time Potential: Without the CRF layer, the ResNet18-based version of the network is incredibly fast, reaching 305 FPS. However, the computationally intensive CRF layer drops the speed to approximately 3.3 - 3.6 FPS, highlighting a trade-off between temporal accuracy and processing speed.
Conclusion and Future Work
MBS-Net proves that modeling the background is a viable and efficient alternative to traditional object detection for autonomous vehicles. By treating background subtraction as a sequence problem rather than a frame-by-frame task, the network achieves superior robustness against camera motion.
Future research will focus on overcoming the computational bottleneck of the CRF layer, potentially using reinforcement learning or more efficient regularization techniques to maintain high accuracy at real-time speeds. Additionally, the authors plan to expand the system into "semantic" background subtraction, where the background is categorized into specific types like "sidewalk" versus "drivable road."
Summary updated