Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient

Shaoqi Wang, Chunjie Yang, Siwei Lou

Abstract

Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor contexts, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of the network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU), which has a solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation (MVE) in NN, wherein the truncated gradient approximates the natural gradient (NG). Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in the soft sensor field.

Introduction

In the world of industrial manufacturing, "soft sensors" are critical mathematical models that estimate difficult-to-measure variables, such as chemical concentrations, using easy-to-measure data like temperature and pressure. While deep learning has revolutionized many fields, its application in industrial soft sensors is often hindered by training instability. In a factory environment, a model that is 99% accurate but occasionally "glitches" or fluctuates during online updates is a safety hazard.

Most neural network research focuses on maximizing offline accuracy. However, industrial settings prioritize online optimization stability and interpretability. Standard training techniques—such as early stopping or complex learning rate schedules—are often impractical for models deployed in real-time control loops. To address this, the Approximated Orthogonal Projection Unit (AOPU) was developed to bring mathematical rigor and rock-solid stability to regression tasks by leveraging the power of Natural Gradients.

Methodology

The AOPU architecture bridges the gap between deep learning and classical estimation theory through three primary mechanisms:

1. Trackable vs. Untrackable Parameters

The authors define "trackable" parameters as those that can be decoupled from input data via a simple inner product. While stacking non-linear activation functions (like ReLU) typically makes parameters "untrackable," AOPU utilizes a robust data augmentation block (consisting of random weights and non-linearities) followed by a final layer of trackable parameters. This structure allows for more precise control over the optimization process.

2. Dual Parameters and Truncated Gradients

AOPU introduces "dual parameters," which are injective representations of the trackable weights. By truncating the gradient backpropagation at these dual parameters, the model effectively approximates the Natural Gradient (NG). This provides the benefits of second-order optimization—which considers the underlying geometry of the parameter space—without the massive computational cost associated with calculating a Fisher Information Matrix.

3. Minimum Variance Estimation and Rank Ratio

The authors prove that AOPU attains Minimum Variance Estimation (MVE), the theoretical "ceiling" for regression performance. To monitor this, they introduce the Rank Ratio (RR), defined as the ratio of linearly independent samples in a batch to the batch size. A high RR indicates the model is closer to the theoretical MVE and will be more numerically stable, providing engineers with a clear interpretability metric for model health.

Experimental Setup

The model was evaluated on two real-world chemical process datasets:

  • Debutanizer Column: Predicting butane content in a desulfurizing plant (2,394 records, 7 sensors).
  • Sulfur Recovery Unit (SRU): Predicting $H_2S$ and $SO_2$ concentrations to prevent environmental pollution (10,080 records, 5 sensors).

AOPU was compared against seven baselines, including LSTMs, Autoformers, Informers, and standard Deep Neural Networks (DNNs).

Results and Discussion

Exceptional Training Stability

In head-to-head tests, AOPU demonstrated almost zero fluctuations in validation loss during training. In contrast, models like the Informer and DNN exhibited significant "spikes" in error, which could be catastrophic in a real-time industrial control loop. By approximating the Natural Gradient, AOPU "recognizes the manifold" of the data, leading to much smoother convergence.

Reliability Without "Tricks"

In industrial deployment, it is often necessary to use the final model checkpoint rather than the "best" one found during offline validation. When forced to use the final checkpoint, the performance of baseline models plummeted. For example, the Autoformer's $R^2$ score dropped by over 120% on the Debutanizer dataset. Conversely, AOPU’s performance remained remarkably consistent, with only a minor decline (1.7% to 14.9%), proving its inherent robustness.

Rapid Convergence

Beyond stability, AOPU reached its optimal performance significantly faster than RNN or Attention-based architectures. This rapid convergence is a direct result of the truncated gradient approach, which optimizes the direction of parameter updates more effectively than standard Gradient Descent.

Conclusion

The AOPU offers a significant advancement for industrial soft sensors by prioritizing stability and mathematical grounding. By truncating gradients to approximate the Natural Gradient, it achieves Minimum Variance Estimation while maintaining the feature extraction benefits of neural networks. While the model currently shows sensitivity to batch size and sequence length relative to feature dimensions, future work will focus on reducing these sensitivities and exploring more sophisticated augmentation blocks to further enhance non-linear modeling capabilities.

Summary updated

Abstract

Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation (MVE) in NN, wherein the truncated gradient approximates the natural gradient (NG). Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.
BibTeX
@inproceedings{wang2024approximated,
  author = {Shaoqi Wang and Chunjie Yang and Siwei Lou},
  title = {{Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=xqrlhsbcwN},
}
Figure