Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient

ShaoQi Wang, Chunjie Yang, Siwei Lou

In the high-stakes world of industrial manufacturing—such as chemical plants or oil refineries—sensors are critical. However, some variables, like the concentration of a specific chemical in a boiling vat, are notoriously difficult to measure in real-time. To bridge this gap, engineers rely on "soft sensors": neural networks that estimate these complex values based on easier-to-measure data like temperature and pressure.

While standard deep learning models excel at offline accuracy, they frequently exhibit dangerous instability during online deployment. A recent paper from researchers at Zhejiang University, titled "Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient," introduces a novel architecture called the AOPU to resolve this critical issue.

The Challenge: Stability in Online Training

In an active factory environment, operators cannot simply "stop and restart" a model or rely on common deep learning tricks like early stopping and adaptive learning rates. Soft sensors must be updated online. If a single mini-batch update causes the model’s output to swing wildly, it could trigger erroneous actions from downstream controllers, leading to severe safety risks or economic losses.

Standard Gradient Descent (GD) is "manifold-blind"—it treats all parameters equally, regardless of how they affect the output distribution. Natural Gradient Descent (NGD), on the other hand, is the gold standard for stability because it accounts for the "information geometry" of the model.

Comparison of optimization trajectories between standard Gradient Descent (GD) and Natural Gradient Descent (NGD).

Unfortunately, NGD requires inverting a massive Fisher Information Matrix (FIM), which is computationally prohibitive for most neural networks. The authors sought a way to achieve the stability of NGD without the crippling computational cost.

Methods: The Approximated Orthogonal Projection Unit (AOPU)

The core innovation of this research is the Approximated Orthogonal Projection Unit (AOPU), a specialized neural network unit designed for industrial regression tasks where training stability supersedes raw speed. Its design rests on three pillars:

  1. Trackable vs. Untrackable Parameters: The authors define "trackable" parameters as those that can be decoupled from input data via a simple inner product. To maintain trackability, they avoid deep stacks of activation functions. Instead, they utilize a "random weight matrix" approach for data augmentation, structurally simplifying the network while maintaining expressive power.
Structural comparison between a standard neural network with untrackable hidden layers (a) and the proposed trackable architecture using data augmentation (b).
  1. Dual Parameters and Gradient Truncation: This is the mathematical engine of the AOPU. The authors introduce a "dual parameter" $D(\tilde{W})$. By truncating the backpropagation process at this dual parameter, the resulting gradient update for the weights effectively approximates the Natural Gradient. It bypasses the FIM complexity entirely through its structural design.
  2. Rank Ratio (RR): The paper introduces a new metric called the Rank Ratio—the ratio of linearly independent samples in a batch. This serves as an interpretability index. A high RR indicates that the model is successfully approximating Minimum Variance Estimation (MVE), acting as a "health check" for engineers before training begins.

Key Insights and Results

The researchers evaluated AOPU on two real-world industrial benchmarks: the Debutanizer Dataset (predicting butane content in a desulfurizing plant) and the Sulfur Recovery Unit (SRU) Dataset (predicting $H_2S$ and $SO_2$ concentrations). The AOPU was compared against heavy hitters like Autoformer, Informer, LSTM, and standard DNNs.

  • The Geometry of Stability: The paper proves that AOPU training is mathematically equivalent to Stochastic Mirror Descent (SMD) using a Mahalanobis distance regularizer, providing a rigorous theoretical foundation for its stability.
  • Unrivaled Online Stability: In online training simulations where early stopping is prohibited, AOPU showed almost zero performance fluctuations. While models like the Autoformer and LSTM saw their $R^2$ scores collapse or turn negative over 40 epochs, AOPU maintained a steady, high $R^2$ (e.g., ~0.82 on the SRU dataset).
  • Robustness to Hyperparameters: AOPU proved far less sensitive to changes in sequence length. In one SRU dataset test, a baseline model's $R^2$ dropped from 0.56 to -0.87 when changing sequence length, whereas AOPU remained consistently optimal.
  • Lower Variance: The standard deviation of AOPU’s results across multiple runs was significantly lower than all baselines, cementing its status as a reliable choice for industrial deployment.

Limitations and Future Work

The authors note that AOPU is not a universal "plug-and-play" solution. Its performance heavily depends on the Rank Ratio. If the RR is too low—often caused by a batch size that is too large relative to the input dimensions—the matrix inversion becomes unstable, risking model divergence.

Future research will likely focus on developing more sophisticated data augmentation modules to boost nonlinear modeling power, reducing the model's sensitivity to batch size, and exploring Natural Gradient optimizations for even more complex augmentation structures.

Paper Citation: Wang, S., Yang, C., & Lou, S. (2024). Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient. 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Summary updated

Abstract

Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation in NN, wherein the truncated gradient approximates the natural gradient. Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.
BibTeX
@inproceedings{wang2024approximated,
  author = {ShaoQi Wang and Chunjie Yang and Siwei Lou},
  title = {{Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=xqrlhsbcwN},
}
Figure