Systematic Development of a New Variational Autoencoder Model Based on Uncertain Data for Monitoring Nonlinear Processes

Kai Wang, Michael G. Forbes, R. Bhushan Gopaluni, Junghui Chen, Zhihuan Song

Introduction

Modern industrial processes, such as chemical manufacturing and paper milling, rely on thousands of sensors to maintain safety and efficiency. Detecting "faults"—deviations from normal operation like equipment failure or process drift—is a critical challenge. These systems are inherently nonlinear and plagued by measurement noise, making traditional linear tools like Principal Component Analysis (PCA) inadequate. While deep learning models like Stacked Denoising Autoencoders (SDAE) can capture nonlinearity, they often function as "black boxes" that provide point estimates, ignoring the underlying uncertainty in the data.

This research proposes a systematic framework for process monitoring using Variational Autoencoders (VAEs). By treating industrial data as a collection of probability distributions rather than static points, the model provides a statistically rigorous approach to detecting subtle anomalies in complex, noisy environments.

Methodology: The Probabilistic Framework

The core of the proposed system is the Variational Autoencoder (VAE). Unlike standard autoencoders that compress data into a single point in a latent space, the VAE maps input data $x_i$ to a probability distribution defined by a mean and variance.

The VAE architecture uses an encoder to define a distribution, samples from it, and then uses a decoder to reconstruct the input, optimizing for both reconstruction accuracy and distribution regularity.

The authors introduced several key technical innovations to adapt VAEs for industrial safety:

  1. Orthogonal Latent Variables: By constraining the covariance matrix of the latent variables to be diagonal, the model is forced to learn independent features. This prevents feature overlap, making it easier for engineers to diagnose which specific process component is failing.
  2. Probabilistic Monitoring Indices: Two new indices were developed for real-time monitoring:
  3. $D$ index (Latent Space): Utilizes Kullback-Leibler (KL) divergence to measure how much the current process state deviates from the learned "normal" distribution.
  4. $R$ index (Residual Space): Based on conditional log-likelihood, this measures the model's ability to reconstruct the observed data, highlighting deviations that the latent space might miss.
  5. Systematic Hyperparameter Tuning: To move away from the "trial-and-error" common in deep learning, the paper provides clear guidelines for selecting latent variable counts and training iterations using early-stopping strategies on validation sets.

Experimental Validation

The model was rigorously tested against traditional methods (PCA, Kernel PCA) and modern deep learning benchmarks (SDAE) using two distinct datasets:

  • Numerical Simulation: A synthetic nonlinear system with Gaussian noise allowed for testing 12 specific fault scenarios, including bias and structural changes.
  • Industrial Paper Machine: A large-scale dataset of 100,000 points from a paper mill, tracking variables like moisture content, dry weight, and machine speed.

The VAE-based approach demonstrated superior performance across the board. In the paper machine dataset, traditional shallow methods suffered from False Alarm Rates (FAR) exceeding 5%, whereas the VAE remained reliably below this threshold. More importantly, the VAE excelled at detecting "noise faults"—cases where the signal mean remains constant but variance increases. For these faults, traditional methods often had a 0% detection rate, while the VAE achieved over 26% detection.

Key Insights: Uncertainty as Information

A fundamental shift in this approach is the realization that uncertainty itself is a valuable diagnostic signal. By modeling data as distributions, the VAE captures the "confidence" of its predictions.

Visualization of the mapping between the latent distribution $p(z)$ and the data space, illustrating how the model captures local uncertainty around the process manifold.

In nonlinear systems, models are essentially "local" descriptions of the training data. When a process moves into a new state due to external fluctuations, the latent space index and the residual index react differently. This distinction is crucial for operators to differentiate between a harmless process shift and a genuine mechanical failure. Furthermore, the enforced orthogonality ensures that each latent variable represents a distinct "mode" of the process, enhancing interpretability.

Conclusion and Future Directions

The proposed VAE framework offers a robust, scalable, and statistically sound alternative to traditional process monitoring. It effectively handles large-scale industrial data where methods like KPCA struggle due to memory constraints.

Currently, the model assumes Gaussian distributions for noise and latent variables. While effective, real-world data often contains outliers that deviate from normality. Future research will explore non-Gaussian distributions, such as the Student’s t-distribution for better outlier handling or the Laplace distribution to encourage model sparsity. These advancements will further refine the ability of deep learning models to ensure safety in the increasingly complex landscape of modern industry.

Summary updated

Abstract

Deep learning models have been applied to industrial process fault detection because of their ability to approximate the complex nonlinear behavior. They have been proven to outperform the shallow neural network models. However, there are no good guidelines on how to build these deep models. Therefore, a good deep model is often constructed through a trial-and-error exercise. It is not easy to interpret the model because of features that do not have any physical interpretation. In addition, latent variables (or features) in a deep model are not independent. This causes features to overlap with each other, resulting in challenges in evaluating distributions of features and designing suitable monitoring indices. Finally, typical deep learning models in process monitoring are used in a deterministic manner and do not automatically provide confidence levels for each decision. In this paper, a variational autoencoder is utilized to develop a framework for monitoring uncertain nonlinear processes. The learned latent variables are guaranteed to be independent (or orthogonal) of each other under a specific optimization objective with constraints. The proposed method provides the density estimates of latent variables and residuals instead of point estimates. The density functions are used to design appropriate indices for monitoring. A simulation example and an industrial paper machine example are presented to validate the effectiveness of the proposed method.
BibTeX
@article{wang2019systematic,
  author = {Kai Wang and Michael G. Forbes and R. Bhushan Gopaluni and Junghui Chen and Zhihuan Song},
  title = {{Systematic Development of a New Variational Autoencoder Model Based on Uncertain Data for Monitoring Nonlinear Processes}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2894764},
}
Figure