Introduction
Shark is a high-performance data analysis engine designed to combine the speed of traditional analytical databases with the fault tolerance and scalability of MapReduce. By building on top of Spark and utilizing in-memory Resilient Distributed Datasets (RDDs), Shark achieves SQL query and machine learning performance up to 100x faster than Apache Hive and Hadoop.
The Challenge: Speed vs. Fault Tolerance
Historically, big data systems were divided into two categories. MapReduce-based systems, such as Hive, provided massive scale and fine-grained fault tolerance, allowing long-running jobs to survive node failures. However, they suffered from high latency due to significant task-launching overhead and frequent disk I/O. Conversely, Massively Parallel Processing (MPP) databases offered high performance but were often brittle; a single node failure typically necessitated restarting the entire query.
Furthermore, modern data science requires a unified approach. Analysts often need to integrate SQL results into complex, iterative machine learning pipelines. Managing these across disparate systems is inefficient and error-prone. Shark addresses this by providing a single, unified engine capable of both high-speed SQL and rich analytics.
System Architecture and Methods
Shark is built on the Spark cluster computing framework, which introduces Resilient Distributed Datasets (RDDs). RDDs are collections of objects partitioned across a cluster that can be persisted in memory. Their primary innovation is "lineage"—a record of the operations used to create them—which enables the system to recompute only the specific data lost during a failure rather than restarting the whole job.
To optimize SQL performance on this architecture, the authors implemented several key features:
- Columnar In-Memory Storage: To avoid the overhead of Java object garbage collection, Shark stores data in memory using compressed, columnar arrays of primitives.
- Partial DAG Execution (PDE): Unlike traditional static optimizers, PDE allows Shark to observe actual statistics of intermediate data during query execution. This enables dynamic re-optimization, such as switching from a shuffle join to a broadcast join on the fly.
- Low-Latency Scheduling: Spark’s scheduler reduces task launch overhead from seconds (in Hadoop) to milliseconds, allowing for thousands of tasks per second.
Evaluation and Datasets
The performance of Shark was evaluated across four diverse workloads:
- Pavlo et al. Benchmark: A 2.1 TB dataset comparing MapReduce with analytical DBMSs.
- TPC-H: Standard industry benchmarks at 100 GB and 1 TB scales.
- Real-world Hive Warehouse: 1.7 TB of video session data and query traces from an industrial partner.
- Machine Learning Dataset: A 100 GB synthetic dataset for testing iterative algorithms like Logistic Regression and K-means.
Key Insights
- The Cost of Task Launching: A significant portion of Hadoop's latency stems from the "engineering" overhead of launching tasks. By slashing this time to 50 milliseconds, Shark can utilize many small tasks to balance workloads and mitigate the impact of "straggler" nodes.
- Dynamic Optimization: In big data environments where data is often unstructured, Shark’s ability to re-plan queries during execution (PDE) proves more effective than static pre-computation.
- Unified Fault Tolerance: Because Shark integrates SQL and Machine Learning within the same RDD lineage, it offers end-to-end fault tolerance. If a failure occurs during a machine learning phase, the system can trace back through the SQL steps to recover the necessary data.
Performance Results
Shark demonstrated substantial performance improvements over existing systems:
- SQL Performance: Shark executed SQL queries up to 100x faster than Hive when data was cached in memory, and maintained a 5–10x advantage even when reading from disk.
- Machine Learning Efficiency: For iterative algorithms like Logistic Regression, Shark outperformed Hadoop by 100x by keeping data in memory between iterations.
- Resilience: In tests where a node was intentionally failed mid-query, Shark recovered and completed the task with only a ~3-second delay, whereas traditional MPP databases would have required a complete restart.
Limitations and Future Directions
Despite its successes, the authors identified several areas for further development:
- JVM Overhead: The Java Virtual Machine still introduces overhead compared to native C++ engines.
- Master Node Vulnerability: While worker nodes are fault-tolerant, the master node remains a single point of failure.
- Automated Caching: Current versions require users to manually specify tables for memory caching; future iterations could automate this based on usage patterns.
Summary updated