Abstract
ReLU (rectified linear units) neural network has received significant attention since its emergence. In this paper, a univariate ReLU (UReLU) neural network is proposed to both modelling the nonlinear dynamic system and revealing insights about the system. Specifically, the neural network consists of neurons with linear and UReLU activation functions, and the UReLU functions are defined as the ReLU functions respect to each dimension. The UReLU neural network is a single hidden layer neural network, and the structure is relatively simple. The initialization of the neural network employs the decoupling method, which provides a good initialization and some insight into the nonlinear system. Compared with normal ReLU neural network, the number of parameters of UReLU network is less, but it still provide a good approximation of the nonlinear dynamic system. The performance of the UReLU neural network is shown through a Hysteretic benchmark system: the Bouc-Wen system. Simulation results verify the effectiveness of the proposed method.
Introduction
In the field of engineering, system identification is the process of building mathematical models of dynamic systems based on observed data. While linear systems are well-understood, most real-world industrial systems exhibit complex nonlinear behavior. Neural networks are powerful tools for this task, but they often suffer from the "black box" problem, where physical insights are difficult to extract, and over-parameterization, which makes them prone to overfitting. This paper introduces the Univariate ReLU (UReLU) neural network to bridge the gap between high-performance modeling and model transparency.
Methods
The core innovation of this work is the UReLU architecture, which simplifies complex nonlinearities by decoupling them into a sum of one-dimensional functions.
The UReLU Architecture
The process begins with a linear transformation matrix $V$ that compresses the input vector (containing past inputs and outputs) into a lower-dimensional intermediate vector $x$. Unlike standard ReLU layers where every neuron processes a weighted sum of all inputs, UReLU neurons are "univariate." Each neuron focuses on just one dimension of $x$ with a specific quantile-based bias. This structure effectively partitions the input space into a grid of polyhedral subregions where the system's behavior is perfectly affine (linear), making it easier for engineers to analyze stability or design controllers.
Initialization via Tensor Decomposition
To address the sensitivity of nonlinear models to starting conditions, the authors utilize a sophisticated tensor decomposition method. By fitting a polynomial model to the data and applying Canonical Polyadic Decomposition (CPD) to its Hessian, the network obtains a "warm start" for the linear transformation $V$, ensuring it focuses on the most critical features of the system from the outset.
Experimental Results
The model was evaluated using the Bouc-Wen Benchmark System, a challenging problem in mechanical engineering used to model hysteresis. The network was trained on 40,960 samples and validated against multi-sine and swept sine inputs to test generalization.
As shown in the performance comparison, the UReLU network achieved state-of-the-art accuracy while maintaining a highly efficient parameter count.
Specifically, UReLU achieved the lowest Root Mean Square Error (RMSE) on both validation sets (-87.18 dB for multi-sine and -96.41 dB for swept sine). Remarkably, it outperformed standard ReLU networks while using nearly 8x fewer parameters (201 vs. 1550), demonstrating superior efficiency and reduced risk of overfitting.
Conclusion
The UReLU neural network provides a streamlined, interpretable alternative for nonlinear system identification. By decoupling nonlinearities and utilizing a supervised dimensionality reduction approach, it achieves high accuracy with minimal parameters. Future work will explore extending this architecture to multi-input, multi-output (MIMO) systems and deeper layers for higher-dimensional datasets.
Summary updated