Industrial Big Data Analytics: Challenges, Methodologies, and Applications

Junping Wang, Wensheng Zhang, Youkang Shi, Shihui Duan, Jin LiuarXiv 2018 · OpenReview

TL;DR

This paper surveys the burgeoning field of Industrial Big Data Analytics, highlighting the critical challenges and proposing a comprehensive reference model to address them. It delves into five key methodologies—data ingestion, repository, management, analytics, and governance—and explores their applications in areas like smart factories and predictive maintenance, aiming to unlock the full potential of manufacturing data.

What Problem Does This Paper Address?

The manufacturing industry is awash in data generated by an ever-increasing number of interconnected systems, devices, and applications. However, this data is often highly distributed, heterogeneous, and comes in various formats, posing significant challenges for effective analysis and real-time decision-making. Traditional data management and analytics approaches are insufficient to handle the "big data era" in manufacturing, which demands new methodologies to manage the volume, velocity, and variety of industrial data. The paper aims to bridge this gap by providing a structured overview of the challenges, proposing solutions, and showcasing the value of industrial big data analytics.

Methods

The paper proposes a five-pronged approach to industrial big data analytics, structured around key methodologies:

  • Highly Distributed Industrial Data Ingestion: Focuses on accessing and integrating data from diverse sources like sensors, devices, and enterprise systems, ensuring data is formatted uniformly.
  • Industrial Big Data Repository: Addresses the need to store and manage massive, heterogeneous datasets, coping with sampling biases and varied data structures.
  • Large-Scale Industrial Data Management: Outlines techniques for organizing, sharing, and managing large volumes of industrial data efficiently.
  • Industrial Data Analytics: Emphasizes tracking data provenance and applying analytical techniques for insights and decision-making.
  • Industrial Data Governance: Ensures data trust, integrity, and security, crucial for sensitive industrial information.

The paper also introduces a reference model for industrial big data analytics, visually depicting these components and their interrelationships. It discusses existing technologies like Apache Hadoop and stream processing frameworks (e.g., Storm) and explores their applicability in the industrial context.

Datasets

Not discussed in the paper. The paper is a survey and conceptual framework, not an empirical study with specific datasets.

Key Insights and Ideas

  • The 5C Architecture: The paper implicitly supports the "5C architecture" (Connection, Conversion, Cyber, Cognition, Configuration) as a necessary step for integrating cyber-physical systems in manufacturing.
  • Beyond Batch Processing: While traditional batch processing (e.g., MapReduce) is foundational, the paper highlights the increasing importance of real-time stream processing for industrial applications where immediate insights are critical.
  • Data Governance is Paramount: The paper strongly emphasizes that for industrial big data to be truly valuable and trustworthy, robust data governance mechanisms for privacy, security, and compliance are non-negotiable.
  • Holistic Value Chain Integration: The true value of industrial big data lies not just in optimizing individual machines or processes but in understanding and improving the entire manufacturing value chain, from suppliers to customers.
  • Bridging the Gap Between Data and Decisions: The ultimate goal is to transform raw data into actionable intelligence that drives better decision-making, leading to increased efficiency, reduced costs, and improved product quality.

Key Results

As a survey paper, it doesn't present quantitative results from experiments. Instead, it synthesizes existing research and industry practices to:

  • Identify and categorize key challenges: Such as lack of spatio-temporal representation, effective online machine learning, and comprehensive data life-cycle management.
  • Propose a structured reference model: Providing a clear framework for understanding and building industrial big data analytics platforms.
  • Highlight successful application areas: Demonstrating the tangible benefits of industrial big data analytics in smart factory visibility, machine fleet optimization, energy management, proactive maintenance, and just-in-time supply chains.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Lack of effective and efficient online large-scale machine learning: This remains a significant challenge, especially with heterogeneous and noisy industrial data.
  • Data visualization: Developing better methods for visualizing massive, multi-modal industrial data is crucial for intuitive understanding and decision-making.
  • Industrial data confidentiality: Ensuring the security and privacy of sensitive industrial data, especially when shared with third parties, requires further development of robust mechanisms.
  • Future work: The paper calls for continued research into developing more effective and efficient industrial big data analytics approaches, particularly in areas like real-time processing, advanced analytics, and robust governance frameworks. The integration of these technologies is seen as key to enabling the future of manufacturing.

Summary updated

Abstract

While manufacturers have been generating highly distributed data from various systems, devices and applications, a number of challenges in both data management and data analysis require new approaches to support the big data era. These challenges for industrial big data analytics is real-time analysis and decision-making from massive heterogeneous data sources in manufacturing space. This survey presents new concepts, methodologies, and applications scenarios of industrial big data analytics, which can provide dramatic improvements in velocity and veracity problem solving. We focus on five important methodologies of industrial big data analytics: 1) Highly distributed industrial data ingestion: access and integrate to highly distributed data sources from various systems, devices and applications; 2) Industrial big data repository: cope with sampling biases and heterogeneity, and store different data formats and structures; 3) Large-scale industrial data management: organizes massive heterogeneous data and share large-scale data; 4) Industrial data analytics: track data provenance, from data generation through data preparation; 5) Industrial data governance: ensures data trust, integrity and security. For each phase, we introduce to current research in industries and academia, and discusses challenges and potential solutions. We also examine the typical applications of industrial big data, including smart factory visibility, machine fleet, energy management, proactive maintenance, and just in time supply chain. These discussions aim to understand the value of industrial big data. Lastly, this survey is concluded with a discussion of open problems and future directions.
BibTeX
@misc{wang2018industrial,
  author = {Junping Wang and Wensheng Zhang and Youkang Shi and Shihui Duan and Jin Liu},
  title = {{Industrial Big Data Analytics: Challenges, Methodologies, and Applications}},
  year = {2018},
  eprint = {1807.01016},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.01016},
}
Figure