Improving a Real-Time Object Detector with Compact Temporal Information

Martin Ahrnbom, Morten Borno Jensen, Kalle Åström, Mikael G. Nilsson, Håkan Ardö, Thomas B. MoeslundWorkshops @ ICCV 2017 · DOI

Introduction

Real-time object detection in video surveillance faces a significant challenge: standard single-frame detectors like SSD or YOLO often fail to capture the temporal context that humans use to identify small, moving, or partially occluded objects. While video-specific architectures like 3D Convolutions or Recurrent Neural Networks (RNNs) exist, they are frequently too computationally expensive for real-time edge deployment.

Researchers have introduced BILSSD (Bonus Input Layer Single-Shot multibox Detector), a novel architecture designed to bridge this gap. By incorporating a compact "foreground probability map" as a fourth input channel, BILSSD provides the model with temporal memory without the heavy overhead of traditional video processing methods.

The BILSSD Framework

The core innovation of BILSSD is the integration of a motion-based feature map alongside standard RGB data. The architecture employs a "deep fusion" approach to process these distinct modalities.

  1. The Fourth Channel (F): In addition to the three RGB channels, the model accepts a foreground probability map generated by BGGRAD, a background subtraction algorithm. This map highlights moving pixels, effectively "pre-digesting" temporal information for the neural network.
  2. Parallel Processing: The network features two parallel input branches. The RGB branch utilizes standard VGG-16 weights, while the Foreground branch uses a smaller, specialized set of layers.
  3. Feature Fusion: These branches are merged at a deeper level before being passed to the final detection layers, allowing the model to weigh visual appearance against motion cues.
The BILSSD pipeline: RGB and foreground maps are fused to produce final object detections.

Training Strategy and Datasets

A significant hurdle in multimodal detection is the lack of massive pre-training datasets for non-RGB inputs. To address this, the authors developed a "Simple Network" pre-training strategy. This ensured the foreground branch learned meaningful localization features from motion data before being integrated into the full model; without this step, the network tended to ignore the motion channel in favor of the more mature RGB features.

The model was evaluated using the VIRAT dataset, a high-resolution surveillance collection. Because VIRAT was originally intended for event recognition, the researchers produced and released new third-party annotations specifically for detecting "vulnerable road users" (pedestrians/cyclists) and "vehicles."

A foreground probability map generated by the BGGRAD algorithm, providing a compact representation of motion.

Key Results and Insights

BILSSD demonstrates that compact temporal data can be more valuable than raw pixel resolution. Key findings include:

  • Significant Accuracy Gains: BILSSD512 achieved a mean Average Precision (mAP) of 0.400, a 66% relative improvement over the RGB-only SSD baseline (0.241).
  • Efficiency: The system maintains real-time performance, with the BILSSD300 variant running at 32 FPS on a Titan X GPU, despite the added complexity of the background subtraction and fusion layers.
  • Contextual Superiority: The model proved particularly adept at identifying small pedestrians and distinguishing between stationary background objects (like parked cars) and active foreground participants.

Conclusion and Future Work

BILSSD offers a practical middle ground for video object detection, providing the benefits of temporal context with minimal latency. However, the current reliance on stationary background subtraction limits its use to fixed-camera surveillance. Future research may explore more robust motion extraction methods for moving platforms or the integration of other "bonus" layers, such as depth or thermal data, to further enhance detection in challenging environments.

Summary updated

BibTeX
@inproceedings{ahrnbom2017improving,
  author = {Martin Ahrnbom and Morten Borno Jensen and Kalle Åström and Mikael G. Nilsson and Håkan Ardö and Thomas B. Moeslund},
  title = {{Improving a Real-Time Object Detector with Compact Temporal Information}},
  booktitle = {Workshops @ ICCV},
  year = {2017},
  doi = {10.1109/iccvw.2017.31},
}
Figure