Big Data Systems Meet Machine Learning Challenges: Towards Big Data Science as a Service

Radwa El Shawi, Sherif Sakr, Domenico Talia, Paolo TrunfioBig Data res 2018 · DOI

TL;DR

This paper explores the burgeoning field of "Big Data Science as a Service," aiming to make sophisticated data analysis accessible to a wider audience. It outlines the essential software stack, from cloud computing resources to machine learning frameworks, and discusses the challenges and opportunities in democratizing data science.

What Problem Does This Paper Address?

The paper addresses the growing challenge of extracting meaningful insights from the ever-increasing volume, velocity, and variety of data being generated today – a phenomenon known as Big Data. While powerful tools and techniques for Big Data analysis exist, they often require deep technical expertise, making them inaccessible to many researchers and practitioners. The authors aim to bridge this gap by proposing and analyzing a "Big Data Science as a Service" model, which leverages cloud computing and existing frameworks to provide these capabilities as a commodity. This democratizes access to data science, enabling individuals and smaller organizations to harness the power of big data without needing extensive infrastructure or specialized knowledge.

Methods

The paper doesn't introduce novel algorithms or techniques but rather focuses on the architectural and service-oriented aspects of Big Data Science. It analyzes the existing software stack, categorizing it into three main cloud service models:

  • Infrastructure as a Service (IaaS): Provides fundamental computing resources like storage and processing power (e.g., Amazon EC2, Google Compute Engine).
  • Platform as a Service (PaaS): Offers a software platform for developing and deploying applications, abstracting away infrastructure management (e.g., Google App Engine, Azure Platform).
  • Software as a Service (SaaS): Delivers complete applications or services accessible via thin clients, with no user management of underlying resources (e.g., Salesforce, Azure ML).

The paper then delves into specific Big Data Science frameworks and tools that operate within these cloud models, including:

  • Statistical and Data Analytics Tools: R, Python.
  • Distributed Big Data Processing Frameworks: Hadoop, Spark, Flink, Storm, Hive.
  • Machine Learning Frameworks: Mahout, SystemML, Azure ML, Amazon ML, TensorFlow, MXNet.

The core "method" is the synthesis and analysis of these existing components to illustrate how they can be combined to offer Big Data Science as a service.

Datasets

The paper does not introduce or evaluate on specific datasets. Instead, it discusses the general need for massive and diverse datasets in Big Data Science and highlights the challenges of data silos and the trend towards open data initiatives and online data markets as potential solutions.

Key Insights and Ideas

  • Big Data Science as a Commodity: The central idea is that by leveraging cloud computing and existing frameworks, Big Data Science can be offered as a utility, much like electricity or water, making it accessible to a broader user base.
  • The "XaaS" Trend: The paper positions Big Data Science as a service within the broader "Everything as a Service" (XaaS) paradigm, emphasizing the shift towards outsourced and managed IT solutions.
  • Democratization of Data Science: By abstracting away the complexities of infrastructure management and algorithm implementation, this service model empowers data scientists and researchers who may not have deep systems engineering expertise.
  • Interoperability is Crucial: As the number of platforms and services grows, the need for standardized formats and models to ensure seamless cooperation becomes paramount.
  • Challenges of Distributed Data: Existing big-data platforms often assume centralized management and homogeneous environments, which can be suboptimal for highly distributed or hybrid data analytics scenarios.

Key Results

This paper is primarily a survey and conceptual analysis, not an empirical study with quantitative results. Therefore, there are no specific numerical results to report. The "results" are the insights and observations about the current state and future potential of Big Data Science as a Service.

Limitations and Future Work

  • Data Silos: A significant challenge identified is the fragmentation of data across different individuals, departments, and organizations, hindering the creation of comprehensive datasets. Future work should focus on motivating collaborative data sharing.
  • Interoperability: The lack of standardized formats and models across various platforms poses a challenge. Future research should aim to develop these standards to facilitate easier cooperation.
  • Distributed Data Challenges: Current big-data platforms are often designed for single-cluster setups. There's a need for more robust solutions that can effectively handle highly distributed data analytics tasks in hybrid environments.
  • Automating Model Discovery: While progress has been made, automating the process of finding the best model or algorithm for a given data challenge remains an active area of research.
  • Lifecycle Management: Efficiently managing the entire lifecycle of data science processes, from data preparation to model deployment and maintenance, requires further research and development.

Summary updated

BibTeX
@article{shawi2018big,
  author = {Radwa El Shawi and Sherif Sakr and Domenico Talia and Paolo Trunfio},
  title = {{Big Data Systems Meet Machine Learning Challenges: Towards Big Data Science as a Service}},
  journal = {Big Data res},
  year = {2018},
  doi = {10.1016/j.bdr.2018.04.004},
}
Figure