SEIZE: Runtime Inspection for Parallel Dataflow Systems
Lists
Only you see this
Abstract
Many Data-Intensive Scalable Computing (DISC) Systems provide easy-to-use functional APIs, and efficient scheduling and execution strategies allowing users to build concise data-parallel programs. In these systems, data transformations are concealed by exposed APIs, and intermediate execution states are masked under dataflow transitions. Consequently, many crucial features and optimizations (e.g., debugging, data provenance, runtime skew detection), which require runtime datafow states, are not well-supported. Inspired by our experience in implementing features and optimizations over DISC systems, we present SEIZE, a unified framework that enables dataflow inspection-wiretapping the data-path with listening logic-in MapReduce-style programming model. We generalize our lessons learned by providing a set of primitives defining dataflow inspection, orchestration options for different inspection granularities, and operator decomposition and dataflow punctuation strategy for dataflow intervention. We demonstrate the generality and flexibility of the approach by deploying SEIZE in both Apache Spark and Apache Flink, and by implementing a prototype runtime query optimizer for Spark. Our experiments show that, the overhead introduced by the inspection logic is most of the time negligible (less than 5 percent in Spark and 10 percent in Flink).
BibTeX
@article{li2021seize,
author = {Youfu Li and Matteo Interlandi and Fotis Psallidas and Wei Wang and Carlo Zaniolo},
title = {{SEIZE: Runtime Inspection for Parallel Dataflow Systems}},
journal = {IEEE Trans. Parallel Distributed Syst.},
year = {2021},
url = {https://openreview.net/forum?id=kxofPJc6Tj4},
}Loading related content...