Abstract
State estimation in highly nonlinear systems with non-Gaussian noise remains a significant challenge for traditional Bayesian filters. This paper proposes the Datatic Approximate Optimal Filter (DAOF), which reformulates the filtering problem as a Markovian Decision Process (MDP) solved via reinforcement learning (RL). DAOF offers two architectures: a model-assisted version (DAOF-v1) and a purely data-driven, model-free version (DAOF-v2). Experimental results on complex vehicle dynamics models show that DAOF significantly outperforms Particle Filters and Supervised Learning Filters in both accuracy and computational efficiency, particularly in the presence of heavy-tailed noise and multimodal dynamics.
Introduction
State estimation, or filtering, is essential for inferring hidden system states from noisy sensor data in modern control systems. While the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF) are standard for linear or mildly nonlinear systems with Gaussian noise, they often fail in highly nonlinear environments or when faced with non-Gaussian noise, such as sensor outliers. Particle Filters (PF) can address these complexities but are computationally intensive, hindering real-time application.
Furthermore, traditional "modelic" methods require precise mathematical models of system dynamics. For complex systems like a 14-degree-of-freedom (DOF) vehicle, deriving these equations is difficult. This paper introduces a "datatic" (data-driven) approach that learns from interaction data, handles non-Gaussian noise gracefully, and maintains low computational overhead.
Methodology: The Markovian Filtering Problem
The authors reframe filtering as a Markovian Filtering Problem (MFP). The objective is to minimize the discounted accumulated estimation error over time using an actor-critic Reinforcement Learning (RL) framework. In this setup, the RL agent's "state" consists of a sliding window of historical sensor measurements and past estimates, while its "action" is the current state estimate.
Two distinct architectures are proposed:
- DAOF-v1 (Model-Assisted): Mimics the predictor-updater structure of a Kalman Filter. A known system model handles the "prediction" step, while the RL policy generates the "update" value.
- DAOF-v2 (Model-Free): Acts as an end-to-end filter, directly outputting state estimates from historical data without requiring explicit transition or observation equations.
The policies are trained offline using the DSAC-T (Distributional Soft Actor-Critic with Three refinements) algorithm. This RL-based approach outperforms standard Supervised Learning Filters (SLF) because it optimizes for an infinite-horizon long-term cost via the Bellman equation, leading to greater stability and better generalization against immediate noise.
Experimental Evaluation
The filters were tested using Monte Carlo simulations of two vehicle dynamics models: a 2-DOF bicycle model and a highly complex 14-DOF model. To simulate real-world conditions, the authors injected non-Gaussian noise, including Gaussian mixture models for process noise and Laplace distributions for measurement noise.
Key Results
In the 2-DOF model, DAOF-v1 achieved a Root Mean Square Error (RMSE) of 0.03 rad for side slip angle, vastly superior to the UKF (39.91 rad) and PF (0.19 rad). Crucially, DAOF-v1 required only 0.65 ms per estimation step, compared to 21.4 ms for the Particle Filter, demonstrating its suitability for real-time deployment.
In the 14-DOF scenario, where traditional Bayesian filters are difficult to formulate, DAOF-v2 successfully tracked all states with near-zero error. The results also highlighted a trade-off: increasing the historical sliding window length improved accuracy but slightly increased computational cost.
Discussion and Conclusion
DAOF demonstrates that RL-based filtering can provide a robust, efficient alternative to traditional methods, especially when mathematical models are unavailable or noise is non-Gaussian. The actor-critic architecture inherently smooths noise by updating the value function through a weighted average of estimates and observations.
A primary limitation is the requirement for "true" state values during offline training. While the filter operates online without ground truth, acquiring training data typically requires high-fidelity simulators or temporary high-precision sensor setups. Future work will focus on physical hardware validation and exploring training methods that utilize partial or noisy ground-truth data.
Summary updated