Filtering Empty Video Frames for Efficient Real-Time Object Detection

Yu Liu, Kyoung‐Don KangSensors 2024 · DOI

TL;DR

This paper introduces L-filter, a novel lightweight method that significantly boosts real-time object detection performance by intelligently filtering out empty video frames. By combining a hybrid time-series analysis with an LSTM model, L-filter achieves higher frame processing rates and improved scalability, enabling more efficient deployment of AI systems in critical applications.

What Problem Does This Paper Address?

Real-time object detection is crucial for many applications, from autonomous driving to surveillance. While deep learning models have revolutionized accuracy, their increasing complexity leads to high latency and resource demands, making true real-time performance a challenge. Specifically, many video streams contain a substantial number of "empty" frames – those without any objects of interest (like vehicles). Processing these frames with computationally expensive object detection models is wasteful and hinders overall system efficiency. The paper aims to bridge this gap by developing a method to efficiently identify and discard these empty frames, thereby accelerating the detection process and enabling systems to handle more data streams simultaneously.

Methods

The core innovation is the L-filter, a lightweight method that acts as a pre-processing step before a standard object detection model. L-filter employs a hybrid approach:

  • Periodic ERD: It periodically runs a lightweight Convolutional Neural Network (CNN) called ERD (Empty Road Detection) – a previous work by the authors – to classify frames as empty or non-empty. This is done less frequently than frame-by-frame analysis.
  • LSTM for Time Series Analysis: For frames between ERD checks, L-filter uses a specially designed Long Short-Term Memory (LSTM) network. This LSTM analyzes a sliding window of recent frame classifications (from ERD) and object counts. By learning temporal patterns and similarities between consecutive frames, the LSTM can predict whether a frame is likely empty or not, without needing to perform dense pixel analysis.

This hybrid approach reduces the computational burden by avoiding full object detection on every frame, while the LSTM's time-series analysis helps maintain high accuracy in predicting empty frames.

Datasets

The paper utilizes two datasets:

  • Dataset 1 (Generated): This dataset was created by manually labeling frames from the DZ Computer Vision dataset. It's notable for having an imbalanced distribution of empty frames (only 21% initially), which the authors addressed with data augmentation to increase the empty frame percentage to 35%. This dataset was used for training the LSTM model.
  • Dataset 2 (Public): This is a public dataset annotated with ground truth bounding boxes for vehicles. Crucially, all frames in this dataset are non-empty. It was used to evaluate the generalizability of L-filter and to assess its worst-case overhead in a heavy traffic scenario.

Key Insights and Ideas

  • Leveraging Temporal Redundancy: The fundamental insight is that consecutive video frames in traffic scenarios are often highly similar, especially regarding the presence or absence of objects. Instead of treating each frame independently, L-filter exploits this temporal redundancy.
  • LSTM for Efficient Prediction: Using an LSTM to analyze a sequence of past classifications (rather than raw pixels) is a clever way to achieve high accuracy in empty frame prediction with significantly lower computational cost.
  • Scalability through Resource Reallocation: By filtering out empty frames, L-filter frees up valuable GPU resources. This allows a single GPU to process multiple video streams concurrently, dramatically improving system scalability and reducing hardware costs.
  • Complementary to Object Detectors: L-filter is designed to be orthogonal to existing object detection models, meaning it can be easily integrated with various SOTA detectors (like YOLOv5, SSD, EfficientDet) to enhance their performance.

Key Results

  • Accuracy: L-filter achieved a road occupancy classification accuracy of 0.99, outperforming ERD and other baselines by 1-18%.
  • Frame Processing Rate (FPS) Enhancement: For a single traffic video stream, L-filter improved the FPS of YOLOv5, SSD, and EfficientDet by 31-47%.
  • Scalability: L-filter, when paired with YOLOv5, can process up to six concurrent video streams on a single commodity GPU, maintaining over 57 FPS per stream. This is a significant improvement over ERD, which could only handle three streams.
  • Overhead: In a worst-case scenario with no empty frames, L-filter introduced a small overhead of 9.38% compared to using YOLOv5 alone.
  • Break-even Threshold: L-filter has a lower break-even threshold for empty frames compared to ERD, meaning it becomes beneficial at a lower percentage of empty frames.

Limitations and Future Work

  • Overhead with Few Empty Frames: L-filter incurs overhead when traffic videos have very few empty frames.
  • Dynamic Activation: The authors suggest dynamically activating L-filter only when traffic patterns suggest a higher likelihood of empty frames, though predicting such patterns accurately is challenging.
  • Integrated Models: Future work could explore integrating L-filter more deeply with object detection models, potentially sharing layers to further reduce overhead.
  • Dataset Limitations: The lack of large-scale, frame-by-frame labeled "empty" vs. "non-empty" datasets is a significant challenge. The authors suggest synthetic data generation as a potential solution.
  • Small Object Detection: The current work focuses on generic object detection; extending it to small object detection is a future research direction.
  • Customizable Object Detection: Allowing users to easily reconfigure the types of objects to detect would be a valuable extension.

Summary updated

Abstract

Deep learning models have significantly improved object detection, which is essential for visual sensing. However, their increasing complexity results in higher latency and resource consumption, making real-time object detection challenging. In order to address the challenge, we propose a new lightweight filtering method called L-filter to predict empty video frames that include no object of interest (e.g., vehicles) with high accuracy via hybrid time series analysis. L-filter drops those frames deemed empty and conducts object detection for nonempty frames only, significantly enhancing the frame processing rate and scalability of real-time object detection. Our evaluation demonstrates that L-filter improves the frame processing rate by 31-47% for a single traffic video stream compared to three standalone state-of-the-art object detection models without L-filter. Additionally, L-filter significantly enhances scalability; it can process up to six concurrent video streams in one commodity GPU, supporting over 57 fps per stream, by working alongside the fastest object detection model among the three models.
BibTeX
@inproceedings{liu2024filtering,
  author = {Yu Liu and Kyoung‐Don Kang},
  title = {{Filtering Empty Video Frames for Efficient Real-Time Object Detection}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24103025},
}
Figure