Recursive State Estimation for Nonlinear Systems

45 concepts42 papers2 proposals

In many engineering and scientific domains, the variables we care about most cannot be observed directly. Whether it is the internal negative electrode potential of an electric vehicle battery, the precise trajectory of a satellite in low Earth orbit, or the latent representation of an image during generative diffusion, we are forced to infer the true "state" of a system from indirect, noisy measurements.

Recursive state estimation solves this problem by continuously updating an internal estimate of the hidden state as new data arrives. Instead of re-processing the entire history of observations at each time step, a recursive estimator maintains a running belief about the system's state and updates it using a structured loop of prediction and correction.

The Core Mechanism: Predict and Correct

To understand why recursive estimation works, imagine tracking a system governed by two sources of information: a physical model of how the system evolves over time, and a stream of sensor measurements.

If we rely solely on our physical model, small modeling errors will accumulate over time, causing our estimate to drift. If we rely solely on our sensors, our estimate will jump wildly due to measurement noise. Recursive estimation elegantly balances these two forces through a two-step cycle:

  1. Prediction: The algorithm uses a dynamic transition model to project the current state estimate forward in time. This step provides a baseline expectation of where the system should be next, but it increases uncertainty because of process noise.
  2. Correction (The Innovation Loop): When a new measurement arrives, the algorithm calculates the innovation—the discrepancy between the actual measurement and what the model predicted. It then scales this innovation to adjust the predicted state, bringing it closer to reality.

As shown by Herui Zha et al. in Zero-Input Compensation Mechanism-Based Linear Estimation for Systems with Multiple Packet Dropouts and Multiplicative Noises, analyzing this innovation sequence allows the estimator to remain optimal even when data packets are dropped or noise is multiplicative. Mathematically, this structure transforms the system into an "innovation form" where the prediction error acts as a direct feedback signal for self-correction, a property formalized by Elie Rouphael et al. in Toward Stochastic Realization Theory for Generalized Linear Switched Systems With Inputs: Decomposition Into Stochastic and Deterministic Components and Existence and Uniqueness of Innovation Form.

 [ Initial State ] ──> ( Predict Next State ) ──> [ Predicted State ]
                                                          │
                                                          ▼
 [ Corrected State ] <── ( Apply Gain ) <── [ Calculate Innovation ]
                                              (Actual vs. Predicted)

This recursive loop is highly versatile. For instance, Oron Sabag and Victoria Kostina leverage Kalman filter recursions in Feedback capacity of Gaussian channels with memory to track hidden channel states at both the transmitter and receiver, maximizing data transmission rates in real time. Even in deep learning, Ziyue Zhang et al. demonstrate in EasyInv: Toward Fast and Better DDIM Inversion that treating diffusion model inversion as a recursive state estimation problem prevents error accumulation without requiring expensive optimization steps.

Conquering Nonlinearity: Sample-by-Sample Linearization

The classic Kalman filter assumes that the system's dynamics and measurement models are strictly linear. However, the physical world is rarely linear. Chemical processes drift, vehicles rotate, and biological systems exhibit complex thresholds.

To handle these realities, the Extended Kalman Filter (EKF) introduces sample-by-sample linearization. Instead of trying to linearize an entire trajectory or signal block upfront, the EKF computes a local linear approximation (via a Jacobian matrix of partial derivatives) around the current state estimate at each individual time step.

Why does this matter? Consider tracking rapid phase fluctuations in telecommunications. As Omar Hazim Salim et al. demonstrate in Channel, Phase Noise, and Frequency Offset in OFDM Systems: Joint Estimation, Data Detection, and Hybrid Cramér-Rao Lower Bound, linearizing over an entire signal block fails to capture fast, non-linear phase noise. By linearizing sample-by-sample, the EKF successfully tracks highly volatile trajectories through state-space.

This local linearization strategy allows the EKF to be deployed across an extraordinary range of non-linear problems:

Managing Memory and Process Drift

A critical challenge in recursive estimation is ensuring the filter remains sensitive to changes in the real world. If the internal model is too rigid, or if the filter places too much confidence in its historical tracking, it can become "blind" to sudden shifts or process drift.

To combat this, researchers modify how the filter handles historical data. For example, Shenhui Du et al. introduce a fading memory factor to the EKF framework in Harmonic Injection Control of Permanent Magnet Synchronous Motor Based on Fading Memory Kalman Filtering. By deliberately discounting older measurements, the filter maintains high responsiveness, allowing it to accurately track motor velocity and suppress torque ripples under changing conditions.

This balancing act between historical filtering and real-time adaptation is a core theme when extracting signals from high-volume, steady-state data. As noted by Alexander Thebelt et al. in Maximizing information from chemical engineering data sets: Applications to machine learning, combining Kalman filtering with moving horizon estimation provides a robust mechanism for filtering out stochastic noise while successfully isolating underlying process drift.

Key Foundational Work

Evidence for variability of atmospheric hydroxyl radicals over the past quarter century (2005) — contributed 1 concept elements to this direction.
Dynamic Bayesian modeling of the cerebral activity (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_recursive_state_estimation_fa17f8ac,
  title = {recursive state estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recursive-state-estimation/rdr_fa17f8ac449d407494ccad1b9c9465b1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_recursive_state_estimation_fa17f8ac_v1,
  title = {recursive state estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recursive-state-estimation/rdr_fa17f8ac449d407494ccad1b9c9465b1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake