Federated cloud-based big data platform in telecommunications

Chao Deng, Ling Qian, Meng Xu, Yujian Du, Zhiguo Luo, Shaoling Sun2012 · DOI

TL;DR

China Mobile, facing massive data growth from its 600 million customers, developed a federated cloud-based big data platform (BC-PDM) to unify data storage, ETL, and data mining. This platform leverages Hadoop and MapReduce on commercial PC clusters to provide a scalable, cost-effective solution for telecommunications analytics and public cloud services.

What Problem Does This Paper Address?

The paper addresses the significant challenge of managing and analyzing the ever-increasing volume of data generated by China Mobile, the world's largest telecommunications operator. With over 600 million customers, the company's IT infrastructure, particularly its data analysis systems, faced bottlenecks in storage and processing power. Traditional, centralized data analysis systems built on expensive UNIX servers struggled to scale and incurred high costs. China Mobile's adoption of a centralized IT strategy based on a federated cloud infrastructure, comprising numerous PC server clusters, exacerbated the big data challenge. To overcome this, they needed a unified, efficient, and cost-effective platform for big data storage, ETL, and data mining that could support their internal analytical needs and enable public cloud services.

Methods

The core of the solution is the BC-PDM (Big Cloud – Parallel Data Mining) platform, built on a federated cloud infrastructure of commercial PC server clusters. Key methodological components include:

  • Hadoop Ecosystem: The platform utilizes Hadoop, specifically HDFS (Hadoop Distributed File System) for distributed storage and MapReduce for parallel processing. This is a standard approach for big data but is applied here within a large-scale telecommunications context.
  • Unified Framework: BC-PDM integrates big data storage, ETL (Extract, Transfer, Load), and Data Mining (DM) into a single, unified framework. This aims to streamline the data analysis pipeline.
  • Parallel Processing: All core operations (ETL, data mining, statistics, SNA) are designed to run in parallel using MapReduce. This is crucial for handling large datasets efficiently.
  • Web-based SaaS Interface: A Web2.0-based SaaS (Software as a Service) mode allows users to access and develop data analysis applications via a web browser, eliminating the need for software installation.
  • Workflow GUI: A graphical user interface (GUI) facilitates application design through drag-and-drop operations, allowing users to build complex workflows by connecting various components.
  • Open Architecture and APIs: The platform is designed with an open architecture, allowing users to develop and add new algorithm components. It also provides Java APIs, WebService APIs, and Command Line APIs for third-party integration and PaaS development.
  • Data Management Features: Includes flexible data management for transfer, access, and remote control of private data and meta-information.
  • Workflow Scheduler: A scheduler allows users to automate the execution of analysis applications under predefined conditions.

Datasets

The paper does not explicitly mention specific datasets used for training or evaluation of the BC-PDM platform itself. However, it discusses the types of data processed, such as:

  • GPRS records: Aggregation of GPRS records for analysis.
  • CDR (Calling Detail Record): Mentioned as a significant data source.
  • Signal data: Another type of telecommunications data.

The paper also mentions future plans to offer free access to large-scale public datasets like Google N-Gram datasets and human gene datasets, suggesting these are potential future data sources for the platform.

Key Insights and Ideas

  • Unified Platform for Telco Big Data: The most significant idea is the creation of a single, integrated platform that addresses the diverse big data needs of a telecommunications giant, from raw data ingestion to complex analytical insights.
  • Cost-Effectiveness of PC Clusters: The paper highlights that using commercial PC server clusters with Hadoop provides a significantly lower-cost alternative to traditional data warehousing and UNIX server solutions, achieving a 1/10th cost reduction for ETL tasks.
  • Scalability is Key: The platform's design emphasizes scalability, with parallel processing and a distributed architecture being central to handling massive datasets.
  • Democratizing Data Analysis: The Web2.0 SaaS interface and user-friendly workflow GUI aim to make sophisticated data analysis tools accessible to a broader range of users within the organization, without requiring deep technical expertise in distributed systems.
  • Enabling Public Cloud Services: BC-PDM is positioned not just for internal use but also as a foundation for China Mobile to offer public cloud computing services, leveraging their existing infrastructure and expertise.

Key Results

  • ETL Performance Improvement: For a specific GPRS records aggregation task that previously took 30 hours on a traditional data warehouse, BC-PDM reduced the processing time to just 2 hours and 5 minutes when deployed on 10 PC servers. This represents a significant performance improvement.
  • Scalability:
    • ETL Operations: Showed excellent scalability, with speedup ratios increasing nearly linearly with the number of nodes (from 32 to 128 nodes).
  • Data Mining Algorithms: Demonstrated acceptable scalability, with some algorithms (FP-growth, PC45, PNaiveBayes) achieving desired scalability.
  • Cost Reduction: The cost of 10 PC servers for BC-PDM was reported to be only 1/10th of the cost of traditional Data Warehouse and Unix servers for similar ETL tasks.

Limitations and Future Work

Limitations:

  • Data Mining Scalability: While ETL operations showed excellent scalability, the paper notes that the scalability of parallel data mining algorithms is "acceptable" rather than excellent, suggesting room for improvement.
  • HDFS Issues: The authors acknowledge observed issues with hard disk errors and I/O exceptions in the HDFS PC server cluster.
  • MapReduce Limitations: The MapReduce framework has limitations in improving the parallel performance of iterative data mining algorithms.

Future Work:

  • Enhanced Functionality: Expanding BC-PDM to include more advanced features like complex statistics operations, web mining, OLAP, reporting services, and BI platform services.
  • Addressing Open Issues: Resolving the noted HDFS and MapReduce limitations.
  • User Incentives Plan: Developing strategies to encourage broader adoption and contribution, including offering free access to public datasets and hosting data analysis competitions.
  • Public Cloud Service Expansion: Further developing BC-PDM to provide public cloud computing services.

Summary updated

Abstract

China Mobile is the biggest telecommunication operator in the world, with more than 600 million customers and an ever increasing information technology (IT). To provide better service to 600 million customers and reduce the cost of IT systems, China Mobile adopted a centralized IT strategy based on cloud computing. The big data issue becomes the most significant challenge to the cloud computing based China Mobile IT structure. This paper presents the China Mobile's big data platform based on the cloud. This platform integrates the big data storage, the development and deployment of big data ETL (Extract, Transfer, Load) and DM (Data Mining) into a unified framework. This big data analysis platform can effectively support the analytical tasks in telecommunications, but it can also help China Mobile provide public cloud computing service. In this paper, we introduce the detailed architecture of China Mobile's platform and discuss its performance.
BibTeX
@misc{deng2012federated,
  author = {Chao Deng and Ling Qian and Meng Xu and Yujian Du and Zhiguo Luo and Shaoling Sun},
  title = {{Federated cloud-based big data platform in telecommunications}},
  year = {2012},
  doi = {10.1145/2378975.2378987},
}
Figure