A Weighted Deep Representation Learning Model for Imbalanced Fault Diagnosis in Cyber-Physical Systems

Zhenyu Wu, Guo Yang, Wenfang Lin, Shuyang Yu, Yang JiSensors 2018 · DOI

TL;DR

This paper introduces a novel deep learning model, wLRCL-D, designed to tackle the critical challenge of imbalanced fault diagnosis in Cyber-Physical Systems (CPSs). By combining Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks with a weighted cost-sensitive loss function and an under-sampling strategy, the model effectively learns from sensor data to accurately identify faults, even when faulty samples are rare.

What Problem Does This Paper Address?

Modern Cyber-Physical Systems (CPSs) generate vast amounts of sensor data, making predictive maintenance and fault diagnosis crucial for preventing failures, saving costs, and ensuring efficient operation. However, traditional fault diagnosis methods face two major hurdles:

  1. Expert Knowledge Dependency: Extracting meaningful features from multi-dimensional sensor data often requires extensive domain expertise to hand-craft features. This process is time-consuming, prone to errors, and may not capture all relevant patterns.
  2. Class Imbalance: In industrial settings, normal operational data is abundant, while data representing specific faults is scarce. This imbalance can severely bias machine learning models, causing them to overlook the minority fault classes and perform poorly on critical fault detection tasks.

This paper aims to address these challenges by developing a data-driven approach that automatically extracts features and effectively handles class imbalance.

Methods

The proposed solution, the weighted Long Recurrent Convolutional LSTM (wLRCL-D) model, is a hybrid deep learning architecture designed to capture both spatial and temporal features from sensor data. Its key components and strategies include:

  • Deep Representation Learning: The model leverages a deep neural network architecture to automatically learn features from raw sensor data, eliminating the need for manual feature engineering.
  • Hierarchical Feature Extraction:
    • 2-Layer CNNs: These layers are used to extract spatial and short-term temporal features from individual time window segments of the sensor data.
  • 2-Layer Inner LSTMs: These layers capture long-term temporal dependencies within each segmented time window, learning high-level patterns.
  • 2-Layer Outer LSTMs: These layers then learn the temporal evolution of these high-level patterns across multiple time window segments, capturing the overall system behavior over time.
  • Data Balancing Strategy: To mitigate the class imbalance problem, the paper employs an under-sampling policy. This strategy reduces the number of samples from the majority (normal) class to create a more balanced dataset for training.
  • Weighted Cost-Sensitive Loss Function: A crucial innovation is the use of a weighted cost-sensitive loss function. This function assigns higher penalties for misclassifying minority (faulty) classes compared to misclassifying majority (normal) classes. The weights are designed to be inversely proportional to the imbalance ratio of each class, ensuring that the model prioritizes learning to detect rare faults.

Datasets

The proposed model was evaluated on the PHM 2015 Challenge datasets. This dataset comprises actual operating condition data from industrial plants, including time series sensor measurements, control signals, and labeled fault events (six types of faults and normal events). The paper specifically focuses on fault types 1-5. The dataset is characterized by a significant imbalance ratio between normal and faulty samples.

Key Insights and Ideas

  • End-to-End Learning for CPS Fault Diagnosis: The wLRCL-D model offers an end-to-end solution, automating feature extraction and fault diagnosis, which is a significant step beyond traditional methods requiring manual feature engineering.
  • Synergy of CNNs and LSTMs for Temporal Data: The hierarchical combination of CNNs for local feature extraction and LSTMs for sequential modeling proves effective in capturing the complex spatio-temporal dynamics inherent in CPS sensor data.
  • The Power of Weighted Loss for Imbalance: The weighted cost-sensitive loss function is a critical component for addressing class imbalance. It directly guides the model to pay more attention to the under-represented faulty classes, leading to improved detection accuracy for these crucial events.
  • Under-sampling as an Effective Balancing Strategy: While over-sampling methods like SMOTE exist, the paper demonstrates that random under-sampling of normal samples, when combined with the weighted loss, is an effective strategy for balancing the dataset and improving model performance.
  • Robustness to Window Length: The proposed wLRCL-D model exhibits greater robustness to variations in the time window length used for segmentation compared to models without the weighted loss, suggesting better generalization capabilities.

Key Results

The wLRCL-D model demonstrated superior performance across various evaluation metrics (precision, recall, and F1-score) compared to several baseline methods, including XGBoost, CNN-D, DeepConvLSTM-D, and other LRCL variants.

  • Overall Performance: The paper reports that wLRCL-D achieved the highest average precision, recall, and F1 scores on the PHM 2015 dataset across different window lengths (24, 48, and 100). For instance, the F1 score for wLRCL-D was consistently high, reaching 98.46% with a window length of 48.
  • Handling Imbalanced Faults: The model showed particular strength in recognizing fault types 4 and 5, which have the most imbalanced ratios, outperforming other methods significantly on these challenging classes.
  • Comparison with Baselines:
    • wLRCL (without the weighted loss) showed improvement over basic LRCL and XGBoost, highlighting the benefit of the weighted loss.
  • Sampling-based methods (like Easy-SMT) generally improved performance over non-sampling baselines.
  • Deep learning methods (like wLRCL-D) outperformed non-deep learning sampling methods.
  • LRCL variants generally showed better feature representation learning than DeepConvLSTM and CNN.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that window length is an important hyper-parameter, and while wLRCL-D is more robust than LRCL-D, its performance can still be influenced by this choice.
  • While effective, the under-sampling strategy might potentially discard some useful information from the majority class.
  • Future Work:
    • The authors suggest exploring more adaptive cost-sensitive weight mechanisms within the loss function to better handle varying imbalance ratios across different datasets or fault types.
  • Investigating hybrid sampling strategies (combining under- and over-sampling) could be a promising avenue.
  • Further research into optimizing the architecture and hyper-parameters of the deep learning model for specific CPS applications could yield even better results.

Summary updated

Abstract

Predictive maintenance plays an important role in modern Cyber-Physical Systems (CPSs) and data-driven methods have been a worthwhile direction for Prognostics Health Management (PHM). However, two main challenges have significant influences on the traditional fault diagnostic models: one is that extracting hand-crafted features from multi-dimensional sensors with internal dependencies depends too much on expertise knowledge; the other is that imbalance pervasively exists among faulty and normal samples. As deep learning models have proved to be good methods for automatic feature extraction, the objective of this paper is to study an optimized deep learning model for imbalanced fault diagnosis for CPSs. Thus, this paper proposes a weighted Long Recurrent Convolutional LSTM model with sampling policy (wLRCL-D) to deal with these challenges. The model consists of 2-layer CNNs, 2-layer inner LSTMs and 2-Layer outer LSTMs, with under-sampling policy and weighted cost-sensitive loss function. Experiments are conducted on PHM 2015 challenge datasets, and the results show that wLRCL-D outperforms other baseline methods.
BibTeX
@inproceedings{wu2018weighted,
  author = {Zhenyu Wu and Guo Yang and Wenfang Lin and Shuyang Yu and Yang Ji},
  title = {{A Weighted Deep Representation Learning Model for Imbalanced Fault Diagnosis in Cyber-Physical Systems}},
  booktitle = {Sensors},
  year = {2018},
  doi = {10.3390/s18041096},
}
Figure