Plug and play bench: Simplifying big data benchmarking using containers

Sheriffo Ceesay, Adam Barker, Blesson Varghese

Abstract

Benchmarking big data frameworks like Hadoop and Spark is often a manual, error-prone, and time-consuming process. Plug and Play Bench (PAPB) addresses these challenges by providing an infrastructure-aware abstraction using Docker containers. PAPB automates the configuration and execution of benchmarks while introducing a novel metric for monetary cost transparency on cloud platforms like Microsoft Azure. This paper details the three-layer architecture of PAPB and evaluates its performance and cost implications across various cluster configurations.

Introduction

The rapid evolution of big data frameworks has led to a proliferation of benchmarking tools, yet researchers still face significant hurdles. First, the fragmentation of benchmark suites makes it difficult to establish standard performance baselines. Second, the deployment complexity of multi-node clusters requires extensive manual configuration of environment variables and memory settings. Finally, existing tools often ignore the financial cost of running workloads in pay-as-you-go cloud environments. PAPB aims to solve these issues by abstracting the infrastructure and providing automated, cost-aware benchmarking.

The PAPB Architecture

PAPB is built on a three-layer architecture designed to integrate seamlessly with existing cluster deployment frameworks, such as the Hortonworks Data Platform (HDP).

The three-layer architecture of PAPB, including the Container, Middleware, and Cluster layers.
  1. Container Layer: This layer packages benchmark tools into Docker images, ensuring environment consistency.
  2. Middleware Layer: Acting as the system's "brains," this layer auto-detects cluster settings and injects them into the containers, removing the need for manual setup.
  3. Cluster Layer: This represents the underlying big data applications (e.g., Spark, Hadoop, NoSQL databases) running on the physical or virtual cluster.

Experimental Evaluation

The authors evaluated PAPB using the HiBench suite, covering 13 workloads across Hadoop and Spark, including micro-benchmarks, machine learning tasks, and SQL analytics. A key finding was the efficiency of "Edge Node" deployment. By mapping host directories directly into containers via Docker volumes, the deployment time was reduced from minutes to just 5 seconds, and image sizes were slashed from 5.2GB to 128MB.

Results: Performance and Cost Analysis

PAPB introduces a Cost Metric formula that correlates execution time with the monthly cost of Azure Virtual Machines. The results demonstrate a clear trade-off: while increasing the number of nodes typically reduces execution time, it simultaneously increases the monetary cost.

Monetary cost analysis across different big data workloads and cluster sizes.

For example, as shown in the cost analysis, a "Hadoop Join" workload saw costs rise from approximately £1.00 on a 4-node cluster to £1.80 on an 8-node cluster. Conversely, performance-heavy tasks like "SparkScan" showed significant time savings, dropping from 75 seconds to 45 seconds when scaling from 4 to 8 nodes.

Conclusion

PAPB simplifies the benchmarking lifecycle by automating setup, data generation, and execution. While currently tailored to HDP and Azure, its modular design allows for future expansion to other cloud platforms and benchmark suites like YCSB. Future work will also focus on refining the cost model to include more complex billing factors such as data egress and spot instance pricing.

Summary updated

BibTeX
@inproceedings{ceesay2017plug,
  author = {Sheriffo Ceesay and Adam Barker and Blesson Varghese},
  title = {{Plug and play bench: Simplifying big data benchmarking using containers}},
  booktitle = {IEEE BigData},
  year = {2017},
  doi = {10.1109/bigdata.2017.8258249},
}
Figure