TL;DR
This paper presents the first implementation of the BigBench benchmark suite on the Hadoop ecosystem, utilizing open-source tools like Hive, Mahout, and NLTK. This work aims to provide a standardized way to evaluate the performance of big data analytics systems, particularly those built on Hadoop, by offering a realistic and comprehensive set of queries and a data model that reflects modern retail analytics use cases.
What Problem Does This Paper Address?
The rapid growth of big data has led to a proliferation of diverse big data management systems (BDMS). However, a significant challenge for users is the lack of standardized, realistic benchmarks to compare the performance and capabilities of these different systems. Existing benchmarks, like TPC-DS, are primarily based on traditional SQL and may not adequately capture the complexities of modern big data analytics, which often involve machine learning, natural language processing, and semi-structured data. BigBench was proposed to address this gap by providing an end-to-end benchmark suite designed for big data analytics, but its implementation was initially limited to proprietary systems. This paper tackles the problem of making BigBench accessible and evaluable within the widely adopted Hadoop ecosystem.
Methods
The authors implemented all 30 BigBench queries within the Hadoop ecosystem, leveraging a combination of open-source technologies:
- Apache Hadoop: Provides the foundational distributed file system (HDFS) for data storage and the MapReduce framework for distributed processing.
- Apache Hive: Acts as a data warehousing solution, offering an SQL-like query language (HiveQL) that translates queries into MapReduce jobs. This allows traditional SQL users to interact with data stored in Hadoop.
- Apache Mahout: A machine learning library that was used to implement queries requiring clustering algorithms, specifically the k-means algorithm for customer segmentation.
- Natural Language Processing Toolkit (NLTK): A Python library used for natural language processing tasks, such as sentiment analysis, integrated into specific BigBench queries.
The implementation involved translating the original BigBench queries (some of which were proprietary SQL-MR) into HiveQL and, where necessary, integrating external programs (like Python scripts for NLTK) via Hive's streaming capabilities.
Datasets
The paper describes the use of synthetically generated data for the BigBench benchmark. This data is based on the TPC-DS schema but with modifications and additions relevant to big data use cases. Specifically, the catalog sales fact table was removed, and dimensions like "Marketprice" and "Web Log" were added. The "Web Log" portion represents semi-structured clickstream data, and "product reviews" represent unstructured data. The authors mention that the data set and queries are available for download.
Key Insights and Ideas
- Bridging the Gap: The successful implementation of BigBench on Hadoop demonstrates that complex, end-to-end big data analytics benchmarks can be realized using widely available open-source tools, making performance evaluation more accessible.
- Flexibility of Hadoop Ecosystem: The paper highlights the flexibility of the Hadoop ecosystem, showing how different components like Hive, Mahout, and NLTK can be integrated to support diverse analytical tasks, from traditional relational queries to advanced machine learning and NLP.
- Challenges in Query Translation: Translating complex queries, especially those involving advanced analytics not native to SQL, into a MapReduce paradigm (via Hive) presents significant engineering challenges. The paper illustrates this with examples of query translation.
- Performance Differences: The evaluation reveals that the reported runtime within Hive/Mahout can differ from the actual Unix time, with the latter often being higher due to system setup and tear-down overhead for each MapReduce job.
Key Results
The paper presents a proof-of-concept evaluation of their BigBench implementation on a 1.3 GB dataset using a single-node setup. The results, presented in a table and a bar chart (Figure 3), show the runtime for each of the 30 BigBench queries.
- Query Runtimes: The table lists "Query System Runtime" (Unix time) and "Query Actual Runtime" (reported by Hive/Mahout) for each query. For example, Query 13 had a system runtime of 4694 seconds and a reported runtime of 4373 seconds.
- JOIN-Heavy Queries: Query 13, which involves numerous JOIN operations, was noted to take significantly longer due to the complexity of translating these joins into MapReduce jobs and the overhead associated with each job.
- General Performance: The Unix time utility generally reported higher times than the internal reporting features of Hive and Mahout, attributed to the setup and tear-down time of MapReduce jobs.
Limitations and Future Work
- Single-Node Setup: The evaluation was conducted on a single-node setup, which does not fully represent the distributed nature of Hadoop. Future work will involve testing on a multi-node cluster.
- Data Generation and Model: The authors plan to improve the data generation process and the data model.
- ETL Pipeline Inclusion: They intend to investigate the inclusion of ETL pipelines, as proposed in other benchmark suites.
- Comprehensive Kit: The team is working on building a complete kit for measuring end-to-end processing time, including loading and refresh operations.
- NLP Model Improvement: The sentiment analysis model, trained on a small dataset, could be improved by training on larger datasets to enhance its repertoire of feature labels.
Summary updated