Performance Evaluation of NoSQL Databases

Andrea Gandini, Marco Gribaudo, William J. Knottenbelt, Rasha Osman, Pietro PiazzollaEPEW 2014 · DOI

TL;DR

This paper benchmarks and models three popular NoSQL databases (Cassandra, MongoDB, and HBase) on Amazon EC2 to understand their performance under various configurations. The authors develop queueing network models that can capture key performance characteristics, providing a basis for capacity planning in cloud environments.

What Problem Does This Paper Address?

As Big Data applications become more prevalent, NoSQL databases are increasingly adopted for their horizontal scalability and schema-free nature. However, a critical challenge for cloud deployments is avoiding overprovisioning, which requires accurately determining the optimal number of nodes for a given workload. This paper addresses the need for a deeper understanding of how different NoSQL databases perform under varying configurations (virtual machine types, cluster sizes) and how this performance can be predicted and modeled for effective capacity planning.

Methods

The authors employ a two-pronged approach:

  • Benchmarking: They use the Yahoo! Cloud Serving Benchmark (YCSB) to stress-test Cassandra, MongoDB, and HBase. Experiments are conducted on Amazon EC2, varying the number of nodes, virtual machine specifications (CPU, memory), and replication factors. Performance metrics like throughput (operations per second) and latency (microseconds per operation) are collected.
  • Queueing Network Modeling: Based on the benchmarking results, they develop high-level queueing network models for each database. These models aim to capture the core performance characteristics, including CPU and disk usage, network latency, and database-specific features like caching and replication. The models are then analyzed using the JMT (Java Modeling Tool) to predict performance.

Datasets

The paper does not use pre-existing datasets. Instead, it generates its own data for the benchmarking experiments using the YCSB framework. YCSB is a standard tool for benchmarking databases, and the authors use it to simulate realistic workloads with a 50% read/50% write mix.

Key Insights and Ideas

  • Database-Specific Performance: Different NoSQL databases exhibit distinct performance characteristics. HBase, for instance, shows strong scaling with the number of cores and generally higher throughput, but its scalability can be less predictable due to non-uniform data distribution.
  • Impact of Configuration: The number of cores per node significantly impacts performance, with most databases showing improvements up to a certain point before plateauing. Replication, while crucial for availability, generally leads to a degradation in throughput as more data copies need to be managed.
  • Modelability of NoSQL: Simple queueing network models can effectively capture many performance aspects of NoSQL databases, especially at higher workloads. This is a significant finding as it suggests a path towards automated capacity planning.
  • HBase Complexity: HBase's architecture, particularly its reliance on Hadoop and its unique data distribution strategy, makes it more challenging to model accurately with simple queueing networks compared to Cassandra and MongoDB.

Key Results

  • Throughput Scaling: Cassandra and MongoDB generally show increasing throughput with more nodes, while HBase also scales well, often outperforming the others in absolute terms.
  • Latency Behavior: Update latency is often I/O bound and shows less variation with core count for Cassandra and HBase. Read latency is more sensitive to parallelization.
  • Replication Overhead: Increased replication factors lead to decreased throughput for Cassandra and MongoDB, with MongoDB showing a less pronounced performance drop. HBase's replication behavior is less clear-cut due to Hadoop's underlying management.
  • Model Accuracy: The queueing network models achieve reasonable accuracy for Cassandra and MongoDB, particularly for throughput and response times at high loads. The model for HBase shows a larger error, especially for throughput, highlighting the challenges in modeling its complex architecture. Relative errors for throughput, write latency, and read latency are reported, generally within acceptable bounds for high loads.

Limitations and Future Work

  • Model Simplification: The authors acknowledge that simple queueing network models cannot perfectly capture all the intricacies of complex NoSQL databases like HBase, especially concerning replication synchronization. The models' accuracy decreases at lower loads.
  • Limited Database Scope: The study focuses on only three popular NoSQL databases.
  • Future Work: The authors suggest extending this work to benchmark other NoSQL databases and developing a more generic modeling framework. Further investigation into modeling complex replication scenarios is also recommended to more accurately reflect the performance of replicated data.

Summary updated

BibTeX
@inproceedings{gandini2014performance,
  author = {Andrea Gandini and Marco Gribaudo and William J. Knottenbelt and Rasha Osman and Pietro Piazzolla},
  title = {{Performance Evaluation of NoSQL Databases}},
  booktitle = {EPEW},
  year = {2014},
  doi = {10.1007/978-3-319-10885-8\_2},
}
Figure