TL;DR
This paper introduces H-Store, a novel distributed main-memory database system designed from the ground up for high-performance Online Transaction Processing (OLTP). By leveraging modern hardware trends like abundant RAM and multi-core processors, H-Store aims to significantly outperform traditional disk-based OLTP systems by optimizing for the specific characteristics of OLTP workloads.
What Problem Does This Paper Address?
The authors argue that traditional Relational Database Management Systems (RDBMSs), many of which are based on architectures developed decades ago (like System R), are no longer optimal for modern OLTP workloads. Several factors contribute to this: the prevalence of multi-core processors, the vast amount of main memory available in servers, the trend towards executing complex logic within stored procedures, and the inherent I/O bottlenecks of disk-based systems. While some existing systems offer distributed capabilities or in-memory storage, they often carry the architectural baggage of older designs. This paper proposes a clean-slate redesign, H-Store, specifically tailored for distributed, in-memory OLTP to overcome these limitations.
Methods
H-Store's architecture is built around several key principles:
- Distributed, Shared-Nothing Architecture: The system is designed to run on a cluster of independent machines (nodes), where each node hosts one or more "sites." Each site is a single-threaded daemon responsible for executing transactions. Crucially, sites on the same machine do not share data structures or memory, simplifying concurrency control.
- In-Memory Storage: All data resides entirely in main memory, eliminating disk I/O as a bottleneck for OLTP operations.
- Stored Procedures as Transactions: OLTP applications interact with H-Store by invoking pre-defined stored procedures. These procedures are compiled into optimized execution plans during a deployment phase.
- Two-Phase Optimization:
- Deployment Time: Stored procedures are compiled into "skeleton plans" using techniques similar to non-distributed databases. This phase optimizes for common query patterns.
- Runtime: On each site, a query planner converts these skeleton plans into multi-site, distributed query plans, taking into account the specific data distribution and network communication costs. A preliminary cost model based on network communications is used.
- Transaction Classes: H-Store identifies and optimizes for specific transaction types:
- Single-sited Transactions: Transactions where all queries can be executed on a single site, minimizing cross-site communication.
- One-shot Transactions: Transactions where individual queries execute on single sites, but their outputs are not used as inputs for subsequent queries within the same transaction.
- Flexible Physical Layout: The system allows for various data partitioning and replication strategies to balance performance and availability. This includes table replication, data partitioning, and k-safety configurations.
Datasets
The paper uses the TPC-C benchmark for evaluation. This is a standard benchmark for OLTP systems, simulating a complex order-entry environment. The authors utilize its transaction set, database schema, and pre-generated data. They highlight that their demonstration system can be configured with different physical database layouts to study their impact on performance.
Key Insights and Ideas
- Clean Slate Advantage: Redesigning an OLTP system from scratch for modern hardware (multi-core, in-memory) offers significant performance advantages over retrofitting existing architectures.
- Stored Procedures are Key: Optimizing for pre-defined stored procedures, rather than ad-hoc queries, is crucial for achieving high OLTP throughput.
- Single-threaded Sites Simplify Concurrency: By making each site single-threaded, H-Store avoids the need for complex concurrent data structures and locking within a site, simplifying the system design.
- Transaction Classification Matters: Identifying and optimizing for specific transaction classes (like single-sited) can lead to substantial performance gains by minimizing network overhead.
- Physical Layout is Critical: The way data is partitioned and replicated has a profound impact on performance and availability in a distributed in-memory system.
Key Results
The paper focuses on the design and demonstration of H-Store. While specific quantitative results are not detailed in this excerpt, the authors state that their earlier prototype "significantly outperform[ed] a traditional, disk-based RDBMS installation using a well-known OLTP benchmark." The demonstration aims to provide insight into the development challenges and allow for further study of the system's behavior under different configurations. They emphasize that the system is designed to outperform legacy OLTP databases by a significant factor.
Limitations and Future Work
- Automatic Database Designer: The current deployment framework assumes a pre-planned database layout. The authors acknowledge that developing an automatic database designer that generates optimized layouts based on schemas and workloads is an area of ongoing research.
- Ad Hoc Query Performance: While H-Store supports ad hoc queries, it provides no guarantees about their execution speed.
- Probabilistic Optimization: The paper mentions plans to investigate probabilistic optimization techniques to infer target sites for queries with unknown parameter values at deployment time.
- Concurrency Control Protocols: The system is currently comparing different optimistic and pessimistic concurrency control protocols for distributed transactions.
Summary updated