hMDAP: A Hybrid Framework for Multi-paradigm Data Analytical Processing on Spark

Xiaowang Zhang, Jiahui Zhang, Zhiyong FengarXiv 2017 · arXiv

Abstract

Modern big data applications increasingly require the simultaneous use of Online Analytical Processing (OLAP), Machine Learning (ML), and Graph Analysis. However, these paradigms are traditionally siloed, leading to data movement overhead and complex development cycles. This paper introduces hMDAP, a hybrid framework built on Apache Spark that unifies these paradigms through an extended relational algebra and a metadata-driven abstraction layer. By providing a single orchestration environment, hMDAP enables seamless multi-stage analytical workflows across heterogeneous data sources.

Introduction

In the era of Big Data, the "variety" of data often poses a more significant challenge than its volume. Applications such as real-time traffic management require structured relational queries, predictive modeling, and network relationship mapping. Traditionally, these tasks necessitate disparate systems—SQL databases for OLAP and specialized libraries like MLlib for modeling—creating fragmented workflows. hMDAP addresses these challenges by offering a unified "one-stop shop" for multi-paradigm processing.

System Architecture

The hMDAP framework is organized into four primary layers designed to abstract complexity and provide a unified interface for diverse analytical tasks.

The high-level architecture of hMDAP, showing the interaction between Query Analysis, Resource Scheduling, Storage Management, and Business Processes.
  1. Storage Management: This layer utilizes HDFS for physical storage but introduces a Logical Storage layer. Through "Abstract Relational Views," it maps heterogeneous data types (e.g., text, RCFile, Sequence files) into a consistent relational format.
  2. Resource Scheduling: Built on Spark, this component leverages MLlib for machine learning and GraphX for graph processing, while integrating MySQL for traditional relational operations.
  3. Query Analysis: Acting as the system's "brain," this layer extends standard SQL/HiveQL syntax with operators for ML and Graph tasks, effectively expanding the relational algebra model to support operations like classification and shortest-path algorithms.
  4. Business Process Module: This module orchestrates the execution flow, splitting complex requests into relational and ML/Graph components.

Workflow Orchestration

A core strength of hMDAP is its ability to manage the lifecycle of a hybrid query. The Business Process Module ensures that data flows correctly between different processing engines.

The hMDAP workflow illustrating how a query is analyzed, split between database and ML tasks, and eventually joined into a final result.

As shown in the workflow, a query (Q) is analyzed and decomposed. For instance, a relational sub-query ($Q_1$) might be executed in a database while a machine learning task ($M_1$) is processed via MLlib. The system handles the training and calculation phases, eventually joining the results from both paradigms into a single output for the user.

Key Insights

  • Unified Algebra: By treating ML and Graph Analysis as extensions of relational algebra rather than external libraries, hMDAP allows for more seamless query optimization.
  • Metadata-Driven Abstraction: The use of "Abstract Relational Views" hides underlying storage complexity, allowing analysts to query semi-structured text files as if they were structured SQL tables.
  • Cost-Based Optimization: The framework employs a cost model using sampling to estimate execution times for mixed-paradigm queries, optimizing resource allocation for hybrid requests.

Results and Limitations

The authors demonstrated hMDAP's functional capabilities through real-world traffic scenarios. Key results include successful interoperability between relational and ML outputs and a user-friendly GUI for managing configurations and viewing results. By building on Spark and HDFS, the framework inherits robust scalability for large-scale distributed environments.

However, some limitations remain. Users must still provide manual XML configuration files for ML parameters, and the paper lacks quantitative performance benchmarks against other hybrid frameworks. Future work should address these configuration overheads and explore real-time latency trade-offs within Spark's batch-oriented architecture.

Summary updated

Abstract

We propose hMDAP, a hybrid framework for large-scale data analytical processing on Spark, to support multi-paradigm process (incl. OLAP, machine learning, and graph analysis etc.) in distributed environments. The framework features a three-layer data process module and a business process module which controls the former. We will demonstrate the strength of hMDAP by using traffic scenarios in a real world.
BibTeX
@misc{zhang2017hmdap,
  author = {Xiaowang Zhang and Jiahui Zhang and Zhiyong Feng},
  title = {{hMDAP: A Hybrid Framework for Multi-paradigm Data Analytical Processing on Spark}},
  year = {2017},
  eprint = {1701.04182},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1701.04182},
}
Figure