Design and Experimental Test of Datatic Approximate Optimal Filter in Nonlinear Dynamic Systems

Weixian He, Zeyu He, Wenhan Cao, Haoyu Gao, Tong Liu, Bin Shuai, Chang Liu, Shengbo Eben Li

Abstract

State estimation in highly nonlinear systems with non-Gaussian noise remains a significant challenge for traditional Bayesian filters. This paper proposes the Datatic Approximate Optimal Filter (DAOF), which reformulates the filtering problem as a Markovian Decision Process (MDP) solved via reinforcement learning (RL). DAOF offers two architectures: a model-assisted version (DAOF-v1) and a purely data-driven, model-free version (DAOF-v2). Experimental results on complex vehicle dynamics models show that DAOF significantly outperforms Particle Filters and Supervised Learning Filters in both accuracy and computational efficiency, particularly in the presence of heavy-tailed noise and multimodal dynamics.

Introduction

State estimation, or filtering, is essential for inferring hidden system states from noisy sensor data in modern control systems. While the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF) are standard for linear or mildly nonlinear systems with Gaussian noise, they often fail in highly nonlinear environments or when faced with non-Gaussian noise, such as sensor outliers. Particle Filters (PF) can address these complexities but are computationally intensive, hindering real-time application.

Furthermore, traditional "modelic" methods require precise mathematical models of system dynamics. For complex systems like a 14-degree-of-freedom (DOF) vehicle, deriving these equations is difficult. This paper introduces a "datatic" (data-driven) approach that learns from interaction data, handles non-Gaussian noise gracefully, and maintains low computational overhead.

Methodology: The Markovian Filtering Problem

The authors reframe filtering as a Markovian Filtering Problem (MFP). The objective is to minimize the discounted accumulated estimation error over time using an actor-critic Reinforcement Learning (RL) framework. In this setup, the RL agent's "state" consists of a sliding window of historical sensor measurements and past estimates, while its "action" is the current state estimate.

The DAOF reinforcement learning framework, illustrating the actor-critic architecture for policy evaluation and improvement.

Two distinct architectures are proposed:

  1. DAOF-v1 (Model-Assisted): Mimics the predictor-updater structure of a Kalman Filter. A known system model handles the "prediction" step, while the RL policy generates the "update" value.
  2. DAOF-v2 (Model-Free): Acts as an end-to-end filter, directly outputting state estimates from historical data without requiring explicit transition or observation equations.

The policies are trained offline using the DSAC-T (Distributional Soft Actor-Critic with Three refinements) algorithm. This RL-based approach outperforms standard Supervised Learning Filters (SLF) because it optimizes for an infinite-horizon long-term cost via the Bellman equation, leading to greater stability and better generalization against immediate noise.

Experimental Evaluation

The filters were tested using Monte Carlo simulations of two vehicle dynamics models: a 2-DOF bicycle model and a highly complex 14-DOF model. To simulate real-world conditions, the authors injected non-Gaussian noise, including Gaussian mixture models for process noise and Laplace distributions for measurement noise.

Key Results

In the 2-DOF model, DAOF-v1 achieved a Root Mean Square Error (RMSE) of 0.03 rad for side slip angle, vastly superior to the UKF (39.91 rad) and PF (0.19 rad). Crucially, DAOF-v1 required only 0.65 ms per estimation step, compared to 21.4 ms for the Particle Filter, demonstrating its suitability for real-time deployment.

In the 14-DOF scenario, where traditional Bayesian filters are difficult to formulate, DAOF-v2 successfully tracked all states with near-zero error. The results also highlighted a trade-off: increasing the historical sliding window length improved accuracy but slightly increased computational cost.

Discussion and Conclusion

DAOF demonstrates that RL-based filtering can provide a robust, efficient alternative to traditional methods, especially when mathematical models are unavailable or noise is non-Gaussian. The actor-critic architecture inherently smooths noise by updating the value function through a weighted average of estimates and observations.

A primary limitation is the requirement for "true" state values during offline training. While the filter operates online without ground truth, acquiring training data typically requires high-fidelity simulators or temporary high-precision sensor setups. Future work will focus on physical hardware validation and exploring training methods that utilize partial or noisy ground-truth data.

Summary updated

Abstract

Filtering is crucial in engineering fields, providing vital state estimation for control systems. However, the nonlinear nature of complex systems and the presence of nonGaussian noises pose significant challenges to the performance of conventional filtering methods in terms of estimation accuracy and computational efficiency. In this work, we present a data-driven closed-loop filter, termed datatic approximate optimal filter (DAOF), specifically designed for nonlinear systems under non-Gaussian conditions. We first formulate a Markovian filtering problem (MFP), which inherently shares a connection with reinforcement learning (RL) as it aims to compute the optimal state estimate by minimizing the accumulated error. To solve MFP, we propose DAOF, which primarily incorporates a trained RL policy and features two distinct structural designs: DAOF-v1 and DAOF-v2. Designed for systems with explicit models, DAOF-v1 combines prediction and update phases, with the RL policy generating the update value. Meanwhile, DAOFv2 bypasses system modeling by directly outputting the state estimate. Then, we utilize an actor-critic algorithm to learn the parameterized policy for DAOF. Experimental results on a 2-degree-of-freedom (2-DOF) vehicle system, equipped with explicit system models, demonstrate the superior accuracy and computational efficiency of DAOF-v1 compared to existing nonlinear filters. Moreover, DAOF-v2 showcases its unique ability to perform filtering without requiring explicit system modeling, as validated by a 14 -DOF vehicle system.
BibTeX
@misc{he2025design,
  author = {Weixian He and Zeyu He and Wenhan Cao and Haoyu Gao and Tong Liu and Bin Shuai and Chang Liu and Shengbo Eben Li},
  title = {{Design and Experimental Test of Datatic Approximate Optimal Filter in Nonlinear Dynamic Systems}},
  year = {2025},
  eprint = {2505.07043},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.07043},
}
Figure