Introduction
Visual change detection (CD) is a cornerstone of computer vision, serving as the primary step for segmenting moving foreground objects from static backgrounds in video streams. It is fundamental to applications ranging from intelligent traffic monitoring and autonomous navigation to high-stakes security surveillance. While the advent of deep learning has propelled the field forward, this review identifies a critical systemic issue: a lack of standardized evaluation protocols that obscures the true progress of the field.
The Evaluation Crisis: SDE vs. SIE
The primary challenge in modern CD research is not just model architecture, but how models are tested. Popular benchmarks like CDnet 2014 provide extensive data but lack officially mandated training and testing splits. This has led researchers to adopt two distinct, and often incomparable, evaluation frameworks:
- Scene Dependent Evaluation (SDE): This setup involves training and testing a model on different frames from the same video sequence.
- Scene Independent Evaluation (SIE): This more rigorous setup requires training on one set of videos and testing on entirely unseen video sequences.
As illustrated in the data division strategies below, SDE allows neural networks to effectively "memorize" the static background geometry and lighting of a specific scene. This leads to inflated performance metrics that do not translate to real-world, unseen environments.
Taxonomy of Deep Learning Frameworks
The review categorizes the current state-of-the-art across several design dimensions:
- Input Modalities: Models vary from single-frame spatial analysis to multi-frame temporal processing (ranging from 2 to 50 frames) to capture motion dynamics.
- Architectural Paradigms: The field has evolved from basic 2D-CNNs (often utilizing pre-trained backbones like VGG16) to more complex 3D-CNNs that process spatial and temporal dimensions simultaneously. Other notable architectures include ConvLSTMs for sequence modeling and Generative Adversarial Networks (GANs) for robust foreground synthesis.
- Hybrid Approaches: Many researchers still rely on "hand-crafted" temporal features (e.g., from algorithms like SuBSENSE) to supplement CNNs, highlighting a lingering difficulty in achieving pure end-to-end spatiotemporal learning.
The Performance Gap
The most significant finding of this empirical review is the massive performance discrepancy between evaluation types. Under SDE conditions, deep learning models frequently report near-perfect F-scores of 98% to 99%. However, these figures are often illusory.
When top-tier models are subjected to SIE, their performance collapses. For instance, the FgSegNetv2 model drops from a 98.9% F-score in SDE to a mere 37.15% in SIE. Currently, the best models specifically optimized for scene independence hover in the 79%–84% range—comparable to the peaks of traditional non-deep learning methods. This suggests that while deep learning has improved background subtraction, the problem of generalized change detection remains far from solved.
Future Research Needs
To move the field toward practical, real-world deployment, the authors advocate for several shifts in research focus:
- Mandatory SIE Benchmarking: The community must adopt standardized, scene-independent splits to ensure models are learning generalizable physics rather than scene-specific backgrounds.
- Unconstrained Motion: Most current datasets assume a static camera. Future work must address "moving-camera" change detection for dashcams and Unmanned Aerial Vehicles (UAVs).
- Efficiency and Latency: Shifting from pixel-wise segmentation to bounding-box detection could facilitate real-time performance on edge devices without sacrificing the utility of the detected changes.
- Self-Supervision: Leveraging the inherent temporal continuity of video through self-supervised learning could reduce the heavy reliance on manually labeled datasets.
Summary updated