In classical control theory, we often assume a "God's eye view" of the world: the controller knows the exact position, velocity, and internal state of the system at every moment. In reality, robots and industrial plants operate in a fog. They rely on noisy sensors, filtered outputs, and incomplete data. This gap between the hidden state (what is actually happening) and the observation (what the sensors report) is the core challenge of Partially Observable Markov Decision Processes (POMDPs).
Recent research has focused on bridging this gap by developing algorithms that can learn to control these systems with the same efficiency and mathematical rigor as if the state were fully visible.
The Predictor Form: Bypassing the Hidden State
The traditional way to handle partial observability is to use a Kalman filter to estimate the hidden state and then apply a controller to that estimate. However, when the system's underlying physics (the dynamics) are unknown, this creates a "chicken and egg" problem: you need the model to estimate the state, but you need the state to learn the model.
A breakthrough in this area, exemplified by the ADAPTON algorithm developed by Sahin Lale, Kamyar Azizzadenesheli, and Animashree Anandkumar, is the use of the predictor form.
Instead of trying to recover the hidden state $x_t$, these methods rephrase the system in terms of observable history. In Logarithmic Regret Bound in Partially Observable Linear Dynamical Systems, the authors show that for linear systems, the relationship between past inputs and past observations can be used to directly predict future observations. By treating the "prediction error" as the new state, researchers can apply reinforcement learning tools directly to observable data, achieving logarithmic regret—meaning the agent stops making sub-optimal mistakes very quickly.
Balancing Exploration and Exploitation in the Dark
In a partially observable world, exploration is dangerous. If you don't know your state, a "wiggle" to learn the dynamics might send the system into an unrecoverable tailspin.
Researchers like Mohamad Kazem Shirani Faradonbeh, Ambuj Tewari, and George Michailidis have tackled this by adapting the principle of Optimism in the Face of Uncertainty (OFU). In Optimism-Based Adaptive Regulation of Linear-Quadratic Systems, they demonstrate that an agent can maintain a set of "plausible" models and choose the one that promises the best performance.
For continuous-time systems governed by diffusion processes, Mohamad Kazem Shirani Faradonbeh and Mohsen Bayati have shown in Thompson Sampling Efficiently Learns to Control Diffusion Processes that Thompson Sampling—randomly sampling a model from a posterior distribution—can stabilize unknown systems with exponentially increasing certainty. This allows for safe learning in high-stakes environments like flight control or medical regulation.
Robustness and Safety via World Models
When the system is nonlinear or the noise is non-Gaussian, simple linear predictors aren't enough. Here, the research shifts toward Model-Based Reinforcement Learning (MBRL) and Robust Control.
- Dreaming Trajectories: In Learning to Fly via Deep Model-Based Reinforcement Learning, Philip Becker-Ehmck and Jan Peters demonstrate how a quadrotor can learn to fly from raw, noisy sensor data. The agent builds a probabilistic "world model" that accounts for uncertainty, allowing it to simulate or "dream" potential maneuvers onboard a low-power processor before executing them.
- Adversarial Resilience: To handle unpredictable environments, Jun Morimoto and Kenji Doya introduced Robust Reinforcement Learning (RRL) in Robust Reinforcement Learning. By framing learning as a "differential game" where an agent plays against an imaginary "disturber," the resulting policies become resilient to the modeling errors inherent in partial observability.
- Implicit Occupancy: Newer methods like LAEO, proposed by Kyle Beltran Hatch and Benjamin Eysenbach in Contrastive Example-Based Control, bypass explicit state estimation entirely. By using contrastive learning to model which states the system is likely to occupy in the future, these agents can maintain control even when sensors are temporarily occluded.
Why This Works: The Power of History
The unifying intuition across these 186 papers is that memory is a proxy for state. Whether through the "predictor form" in linear theory, recurrent architectures in deep RL (like the RD2 agent by Jieliang Luo in Recurrent Distributed Reinforcement Learning for Partially Observable Robotic Assembly), or symbolic differential equations discovered via genetic programming (Sigur de Vries), the solution to partial observability is to leverage the temporal sequence of data.
By treating a sequence of imperfect measurements as a single, high-dimensional "information state," researchers have transformed the "unsolved problem" of POMDPs into a tractable challenge for modern adaptive control.