Realizing Video Analytic Service in the Fog-Based Infrastructure-Less Environments

Qiushi Zheng, Jiong Jin, Tiehua Zhang, Longxiang Gao, Yong Xiang

TL;DR

This paper proposes a novel fog-based video analytics framework designed for infrastructure-less environments, like national parks, where cloud connectivity is unreliable. By intelligently extracting key frames and filtering similar ones using a Siamese neural network, the framework significantly reduces computational load, enabling efficient video analysis on resource-constrained fog devices.

What Problem Does This Paper Address?

The proliferation of deep learning, particularly Convolutional Neural Networks (CNNs), has revolutionized video analytics, enabling sophisticated visual recognition and analysis. However, these powerful models are computationally intensive, typically requiring robust cloud infrastructure for execution. This presents a significant challenge for "infrastructure-less environments" such as remote national parks, rural farms, or disaster zones, where reliable internet and power are scarce. In these settings, traditional cloud-dependent solutions are impractical. The paper aims to bridge this gap by enabling intelligent video analytics directly on edge devices (fog nodes) within these challenging environments, facilitating real-time monitoring and data collection without constant cloud reliance.

Methods

The proposed framework tackles the computational burden of video analytics in two primary stages before final classification:

  1. Key Information Extraction: Instead of processing every frame from a video stream (typically 24-30 fps), the system first identifies "key frames." This is achieved using a frame difference method, which detects significant changes between consecutive frames. This effectively filters out static scenes and focuses on moments with movement, drastically reducing the volume of data. The authors note that the interval between frames can be dynamically adjusted to balance processing speed and power consumption.
  2. Information Filtering (Siamese Neural Network): Even after key frame extraction, many frames might contain similar information (e.g., the same animal in slightly different poses). To further reduce redundancy, a Siamese neural network is employed. This network learns to differentiate between two input images. If two frames are deemed to contain the same animal, one is discarded, retaining only the essential information. This is a departure from traditional CNNs that classify images; the Siamese network's output is simply "same" or "different."
  3. Final Classification (MobileNetV2): For the remaining critical frames, a lightweight CNN, MobileNetV2, is used for the final classification. MobileNetV2 is chosen for its efficiency and suitability for resource-constrained devices, offering a good balance between accuracy and computational cost.

The overall architecture is visualized in Figure 1, showing the flow from video stream to key information extraction, filtering, and finally, classification.

Datasets

  • Training Data: The paper mentions using a dataset of 25,000 dog and cat images from Kaggle for training the MobileNetV2 model. They also generated 30,000 pairs of training data and 6,000 pairs of verification data using pictures of 5 different birds from an observation video to train the Siamese network.
  • Evaluation Data: For evaluating the Siamese network's robustness on unseen classes, 1757 consecutive pictures of pet dog activity from a 4-minute video were used. For experimental evaluation on the Raspberry Pi, animal videos (specifically mentioning dog and cat videos) were downloaded from YouTube and pre-processed to simulate surveillance camera output (720p, 25fps).

Key Insights and Ideas

  • The "Infrastructure-Less" Challenge is Real: The paper effectively highlights the practical limitations of applying cloud-dependent AI in environments with poor or no connectivity, a growing concern with IoT deployments in remote areas.
  • Frame Difference is a Smart First Step: Using frame differencing as an initial filter is a computationally cheap yet effective way to drastically reduce the data volume for video analytics. It's a pragmatic approach to tackle the sheer amount of data generated by video.
  • Siamese Networks for Redundancy Reduction: Employing a Siamese network for similarity comparison is an innovative way to further prune the data. Instead of just identifying objects, it focuses on identifying unique observations, which is crucial for efficiency.
  • MobileNetV2 is a Good Fit for Fog: The choice of MobileNetV2 underscores the importance of selecting models specifically designed for edge and mobile devices, balancing performance with resource constraints.
  • Flexibility is Key: The framework's adaptability to varying computing capabilities and power levels (e.g., battery life) is crucial for real-world deployment in unpredictable environments.

Key Results

  • Significant Frame Reduction: The paper demonstrates a substantial reduction in the number of frames processed. For instance, out of an initial 45,675 frames from 6 videos, only 953 frames (approximately 2.08%) were passed to the final classification stage after frame difference and Siamese network filtering.
  • Siamese Network Accuracy: The Siamese network achieved up to 99.4% accuracy on trained bird species and demonstrated over 75% accuracy on untrained animal classes, showing good generalization potential.
  • MobileNetV2 Accuracy: MobileNetV2 achieved a high accuracy rate of around 96.07% when trained on a dog and cat dataset using transfer learning.
  • Performance on Raspberry Pi: The framework showed satisfactory performance even when operating at the CPU's minimum power state, achieving a better processing frame rate than Yolo-tiny. Figure 4 illustrates processing speeds under different working conditions.

Limitations and Future Work

  • Siamese Network Robustness: While the Siamese network showed promise, its accuracy on untrained animal classes (75%) suggests room for improvement, especially for diverse wildlife monitoring. The authors acknowledge that the high accuracy on trained data was limited to the specific bird species used.
  • Power Consumption Evaluation: The paper mentions that a long-term operation evaluation under different power consumption states is needed. Figure 4 provides a glimpse, but a more in-depth analysis would be beneficial.
  • Communication Strategy: The authors identify their next phase of research will focus on developing a communication strategy to dynamically assign tasks among fog nodes in real-time, aiming for even better processing capabilities. This is crucial for scaling the solution and handling distributed tasks.

Summary updated

Abstract

Deep learning has unleashed the great potential in many fields and now is the most significant facilitator for video analytics owing to its capability to providing more intelligent services in a complex scenario. Meanwhile, the emergence of fog computing has brought unprecedented opportunities to provision intelligence services in infrastructure-less environments like remote national parks and rural farms. However, most of the deep learning algorithms are computationally intensive and impossible to be executed in such environments due to the needed supports from the cloud. In this paper, we develop a video analytic framework, which is tailored particularly for the fog devices to realize video analytic service in a rapid manner. Also, the convolution neural networks are used as the core processing unit in the framework to facilitate the image analysing process.
BibTeX
@inproceedings{zheng2020realizing,
  author = {Qiushi Zheng and Jiong Jin and Tiehua Zhang and Longxiang Gao and Yong Xiang},
  title = {{Realizing Video Analytic Service in the Fog-Based Infrastructure-Less Environments}},
  booktitle = {Fog-IoT},
  year = {2020},
  doi = {10.4230/oasics.fog-iot.2020.11},
}

Loading related content...

Figure