TL;DR
This paper introduces Healthcare4VideoStorm, a framework designed to enhance the dependability of real-time video stream processing applications running on Apache Storm. By monitoring fine-grained metrics and intelligently scheduling resources at the tuple level, it aims to overcome the limitations of existing coarse-grained, network-centric scheduling approaches, leading to improved application performance and reliability.
What Problem Does This Paper Address?
Real-time video stream processing, crucial for applications like security surveillance and smart city initiatives, generates massive amounts of data that demand efficient and reliable processing. Apache Storm is a popular framework for such tasks, but its default scheduling mechanisms are too coarse-grained and primarily focus on network resources, neglecting crucial computing resources. This can lead to performance bottlenecks, especially in complex video analysis pipelines, ultimately impacting the dependability of these applications. Existing solutions often involve rescheduling entire topologies, which is too slow for real-time scenarios. The paper addresses the need for a more granular, resource-aware scheduling approach that can dynamically adapt to changing application states and resource availability.
Methods
The core of the proposed solution is the Healthcare4VideoStorm framework, which comprises four key components:
- Metric Sensor: This component collects detailed, fine-grained metrics from individual Storm tasks, going beyond what Storm's built-in APIs provide. It leverages custom spout and bolt implementations and a dedicated consumer to gather data like CPU utility, latency, throughput, and ack counts.
- Bottleneck Detector: This component analyzes the collected metrics to identify performance degradation. It employs an unsupervised approach using Kernel Density Estimators (KDE) to build probability distributions for metric pairs (e.g., latency-CPU utility, ack count-throughput). A Fuzzy Inference System (FIS) then uses these probabilities to determine the likelihood of a bottleneck.
- Router: Unlike traditional schedulers that operate at the topology level, the router works at the tuple level. It aims to optimize network load by intelligently routing tuples to tasks within the same worker or node whenever possible, minimizing inter-node communication. This is achieved by implementing a custom
RouteGroupingthat considers task locations. - CPU-GPU Switcher: This component addresses computing resource bottlenecks. It monitors the Frames Per Second (FPS) of a task and, if it falls below a required threshold, switches workloads from the CPU to the GPU to accelerate processing. This leverages the parallel processing capabilities of GPUs for computationally intensive video tasks.
Datasets
The paper does not explicitly mention using pre-existing datasets for training or evaluation. Instead, it describes an experimental setup involving custom-designed topologies:
- Target Topology: This topology simulates a video processing pipeline with components for receiving and decoding video streams (
VideoSpout), background subtraction (BgsBolt), and feature extraction (FeatureBolt). - Bottleneck Makers: Two types of topologies were used to induce bottlenecks:
- CPU-intensive topology: This topology performs computationally demanding tasks like finding maximum primes.
- I/O-intensive topology: This topology generates a high frequency of tuples with minimal application-level work, primarily stressing network resources.
The evaluation focuses on the performance of the proposed framework under these controlled bottleneck scenarios.
Key Insights and Ideas
- Fine-grained, Tuple-Level Scheduling is Crucial: The paper strongly advocates for scheduling at the tuple level rather than the topology level. This is because topology-level rescheduling is too slow for real-time applications, while tuple-level routing can dynamically optimize resource utilization without significant downtime.
- Holistic Resource Consideration (CPU & Network): A key contribution is the simultaneous consideration of both network and computing resources. Previous approaches often focused on one or the other, but real-time video processing demands a balanced approach.
- Unsupervised Anomaly Detection for Bottlenecks: The use of an unsupervised approach (KDE and FIS) for bottleneck detection is practical for real-world scenarios where labeled data for every possible topology and task is infeasible to obtain.
- GPU Offloading for CPU Bottlenecks: The CPU-GPU switcher demonstrates the effectiveness of leveraging GPU acceleration to mitigate CPU-bound performance issues in video processing.
- Distinguishing Network vs. Compute Bottlenecks: The evaluation highlights that the router is effective for network bottlenecks, while the CPU-GPU switcher is designed for compute bottlenecks, suggesting a complementary approach to address different types of performance issues.
Key Results
- Router Effectiveness: The router successfully triggered and helped mitigate performance degradation in CPU-intensive and I/O-intensive scenarios. In the I/O-intensive case, the router's triggering was associated with an increase in CPU-throughput probability, indicating a shift in resource utilization.
- CPU-GPU Switcher Performance: The CPU-GPU switcher demonstrated its ability to maintain or recover FPS when bottlenecks occurred. Switchable tasks generally outperformed non-switchable tasks, especially when the FPS dropped below the required threshold. For instance, in the CPU-intensive case, switchable tasks showed a noticeable improvement after the switcher was triggered.
- Discussion on FPS Deviation: The paper discusses that while the processing frame rate (PFR) might show high deviation, the sampling frame rate (SFR) remains constant, which is what truly matters for temporal video analytic algorithms. This suggests that the observed deviations in PFR due to dynamic switching do not necessarily hinder the performance of such algorithms.
Limitations and Future Work
- Memory-Intensive Topologies Not Considered: The authors explicitly state that they did not consider memory-intensive topologies due to the risk of "out of memory" exceptions when forcing memory increases.
- Anomaly Detection Techniques: The paper suggests exploring other anomaly detection techniques beyond the Fuzzy Inference System, such as deep learning and Bayesian models, to potentially improve bottleneck detection.
- Memory Consumption as a Metric: Future work includes extending the framework to consider memory consumption alongside network and computing resources for more comprehensive decision-making.
- Automated Anomaly Detection: The authors plan to explore techniques for automatically detecting anomalies and comparing their performance.
Summary updated