Abstract
Data generation is a key issue in big data benchmarking that aims to generate application-specific data sets to meet the 4V requirements of big data. Specifically, big data generators need to generate scalable data (Volume) of different types (Variety) under controllable generation rates (Velocity) while keeping the important characteristics of raw data (Veracity). This gives rise to various new challenges about how we design generators efficiently and successfully. To date, most existing techniques can only generate limited types of data and support specific big data systems such as Hadoop. Hence we develop a tool, called Big Data Generator Suite (BDGS), to efficiently generate scalable big data while employing data models derived from real data to preserve data veracity. The effectiveness of BDGS is demonstrated by developing six data generators covering three representative data types (structured, semi-structured and unstructured) and three data sources (text, graph, and table data).
Introduction
Benchmarking modern big data systems—ranging from Hadoop and Spark to various NoSQL databases—requires massive, high-fidelity datasets. However, researchers frequently encounter three significant obstacles: privacy concerns that prevent companies from sharing real-world data, the logistical nightmare of transferring petabytes of data over the internet, and the "Veracity Gap." This gap occurs when synthetic data is too simplistic, failing to reflect the complex statistical distributions of real-world information. If a benchmark uses data that lacks realistic internal structures, the resulting performance metrics may not translate to actual production environments.
To address these challenges, we introduce the Big Data Generator Suite (BDGS), a unified tool designed to generate structured, semi-structured, and unstructured data that maintains the "4V" characteristics: Volume, Variety, Velocity, and Veracity.
System Architecture and Methodology
The BDGS architecture is designed as a modular pipeline that transforms real-world "seed" data into massive synthetic workloads. As shown in Figure 1, the process consists of four primary stages: Data Selection, Data Processing (Modeling), Data Generation, and Format Conversion.
The core innovation of BDGS lies in its modeling phase, which ensures data veracity across different formats:
- Unstructured Text: Instead of generating random characters, BDGS utilizes Latent Dirichlet Allocation (LDA). By training on real corpora, it learns topic and word distributions to produce new documents with realistic statistical properties.
- Graph Data: To simulate social networks or web links, the suite employs the Kronecker graph model. This captures essential patterns like "Power Law" distributions and the "Shrinking Diameter" effect observed in growing real-world networks.
- Structured and Semi-structured Data: For relational tables, BDGS integrates the Parallel Data Generation Framework (PDGF). For semi-structured, schema-less data (e.g., resumes), it uses a Bernoulli distribution to model the presence or absence of optional fields.
Seed Datasets
To ensure the generated data "looks" like the real thing, BDGS uses six representative real-world datasets as seeds for its models. These cover a wide variety of sources and types, as detailed in the table below.
These seeds allow the suite to generate data ranging from unstructured Wikipedia articles and Google web graphs to structured e-commerce transaction tables.
Key Insights and Performance
A central tenet of BDGS is that volume alone is insufficient for benchmarking; veracity is the critical requirement. By using model-based scaling, BDGS can learn the underlying parameters of a 1GB sample and generate a 100GB or 1TB file that shares the same statistical "DNA."
Experimental results demonstrate that BDGS achieves linear scalability, meaning generation time grows proportionally with data volume. This is essential for petabyte-scale testing. On a single node, text generation from the Wikipedia model averaged 63.23 MB/s, while Amazon movie reviews reached 71.3 MB/s. Generating 1 TB of Wikipedia-like text takes approximately 4.7 hours.
Conclusion and Future Work
BDGS provides a scalable solution for generating high-veracity big data. While current results show strong linear performance, future work will focus on addressing memory bottlenecks in graph generation and implementing a fully parallelized version for distributed environments. Additionally, the suite will be expanded to include multimedia data types such as images, video, and audio.
Summary updated