Shark: SQL and Rich Analytics at Scale

Reynold Xin, Josh Rosen, Matei Zaharia, Michael J. Franklin, Scott Shenker, Ion StoicaarXiv 2012 · arXiv

Introduction

Shark is a high-performance data analysis engine designed to combine the speed of traditional analytical databases with the fault tolerance and scalability of MapReduce. By building on top of Spark and utilizing in-memory Resilient Distributed Datasets (RDDs), Shark achieves SQL query and machine learning performance up to 100x faster than Apache Hive and Hadoop.

The Challenge: Speed vs. Fault Tolerance

Historically, big data systems were divided into two categories. MapReduce-based systems, such as Hive, provided massive scale and fine-grained fault tolerance, allowing long-running jobs to survive node failures. However, they suffered from high latency due to significant task-launching overhead and frequent disk I/O. Conversely, Massively Parallel Processing (MPP) databases offered high performance but were often brittle; a single node failure typically necessitated restarting the entire query.

Furthermore, modern data science requires a unified approach. Analysts often need to integrate SQL results into complex, iterative machine learning pipelines. Managing these across disparate systems is inefficient and error-prone. Shark addresses this by providing a single, unified engine capable of both high-speed SQL and rich analytics.

System Architecture and Methods

Shark is built on the Spark cluster computing framework, which introduces Resilient Distributed Datasets (RDDs). RDDs are collections of objects partitioned across a cluster that can be persisted in memory. Their primary innovation is "lineage"—a record of the operations used to create them—which enables the system to recompute only the specific data lost during a failure rather than restarting the whole job.

The architecture of a Shark cluster, illustrating the interaction between the Master node and Slave nodes containing Spark Runtimes and Memstores.

To optimize SQL performance on this architecture, the authors implemented several key features:

  • Columnar In-Memory Storage: To avoid the overhead of Java object garbage collection, Shark stores data in memory using compressed, columnar arrays of primitives.
  • Partial DAG Execution (PDE): Unlike traditional static optimizers, PDE allows Shark to observe actual statistics of intermediate data during query execution. This enables dynamic re-optimization, such as switching from a shuffle join to a broadcast join on the fly.
  • Low-Latency Scheduling: Spark’s scheduler reduces task launch overhead from seconds (in Hadoop) to milliseconds, allowing for thousands of tasks per second.

Evaluation and Datasets

The performance of Shark was evaluated across four diverse workloads:

  1. Pavlo et al. Benchmark: A 2.1 TB dataset comparing MapReduce with analytical DBMSs.
  2. TPC-H: Standard industry benchmarks at 100 GB and 1 TB scales.
  3. Real-world Hive Warehouse: 1.7 TB of video session data and query traces from an industrial partner.
  4. Machine Learning Dataset: A 100 GB synthetic dataset for testing iterative algorithms like Logistic Regression and K-means.

Key Insights

  • The Cost of Task Launching: A significant portion of Hadoop's latency stems from the "engineering" overhead of launching tasks. By slashing this time to 50 milliseconds, Shark can utilize many small tasks to balance workloads and mitigate the impact of "straggler" nodes.
  • Dynamic Optimization: In big data environments where data is often unstructured, Shark’s ability to re-plan queries during execution (PDE) proves more effective than static pre-computation.
  • Unified Fault Tolerance: Because Shark integrates SQL and Machine Learning within the same RDD lineage, it offers end-to-end fault tolerance. If a failure occurs during a machine learning phase, the system can trace back through the SQL steps to recover the necessary data.

Performance Results

Shark demonstrated substantial performance improvements over existing systems:

  • SQL Performance: Shark executed SQL queries up to 100x faster than Hive when data was cached in memory, and maintained a 5–10x advantage even when reading from disk.
  • Machine Learning Efficiency: For iterative algorithms like Logistic Regression, Shark outperformed Hadoop by 100x by keeping data in memory between iterations.
  • Resilience: In tests where a node was intentionally failed mid-query, Shark recovered and completed the task with only a ~3-second delay, whereas traditional MPP databases would have required a complete restart.
Performance comparison of Shark versus Hive across SQL queries and Logistic Regression, highlighting Shark's sub-second execution times.

Limitations and Future Directions

Despite its successes, the authors identified several areas for further development:

  • JVM Overhead: The Java Virtual Machine still introduces overhead compared to native C++ engines.
  • Master Node Vulnerability: While worker nodes are fault-tolerant, the master node remains a single point of failure.
  • Automated Caching: Current versions require users to manually specify tables for memory caching; future iterations could automate this based on usage patterns.

Summary updated

Abstract

Shark is a new data analysis system that marries query processing with complex analytics on large clusters. It leverages a novel distributed memory abstraction to provide a unified engine that can run SQL queries and sophisticated analytics functions (e.g., iterative machine learning) at scale, and efficiently recovers from failures mid-query. This allows Shark to run SQL queries up to 100x faster than Apache Hive, and machine learning programs up to 100x faster than Hadoop. Unlike previous systems, Shark shows that it is possible to achieve these speedups while retaining a MapReduce-like execution engine, and the fine-grained fault tolerance properties that such engines provide. It extends such an engine in several ways, including column-oriented in-memory storage and dynamic mid-query replanning, to effectively execute SQL. The result is a system that matches the speedups reported for MPP analytic databases over MapReduce, while offering fault tolerance properties and complex analytics capabilities that they lack.
BibTeX
@misc{xin2012shark,
  author = {Reynold Xin and Josh Rosen and Matei Zaharia and Michael J. Franklin and Scott Shenker and Ion Stoica},
  title = {{Shark: SQL and Rich Analytics at Scale}},
  year = {2012},
  eprint = {1211.6176},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1211.6176},
}
Figure