In standard reinforcement learning and automated planning, we often assume an agent possesses perfect clarity: at any given moment, it knows the exact state of its environment. This ideal setup is modeled as a Markov Decision Process (MDP). However, real-world deployment quickly shatters this assumption. A drone navigating a cluttered forest sees only what is directly in front of its camera; a home assistant robot cannot know if a cup is inside a cupboard without opening it; a medical system must treat a patient based on sparse, noisy lab results.
These scenarios represent partial observability—a challenge where the true state of the world is hidden, and the agent must make sequential decisions based on an incomplete stream of noisy observations. If an agent blindly treats its immediate observations as the absolute state, it suffers from perceptual aliasing: two completely different situations look identical, leading to catastrophic failures. To succeed, the agent must maintain an internal representation of what it cannot see.
Tracking the Invisible via Belief States
How do we make decisions when we don't know where we are? The classical mathematical answer is the Partially Observable Markov Decision Process (POMDP). Instead of mapping a single state to an action, a POMDP maps a belief state—a probability distribution over all possible true states—to an action.
Tracking this distribution explicitly is computationally brutal. As explored in early foundational work like Decision-Theoretic Planning: Structural Assumptions and Computational Leverage by Boutilier et al., managing uncertainty exponentially compounds the complexity of planning. To make this tractable, modern approaches embed state-estimation directly into the planning architecture.
For instance, in QMDP-Net: Deep Learning for Planning under Partial Observability, Karkus et al. introduced a neural network architecture that embeds a differentiable POMDP solver. Instead of separating state estimation (filtering) from policy execution, QMDP-Net learns them jointly end-to-end. This allows the network to maintain a latent belief state that updates dynamically with each new observation. Similarly, for deterministic worlds, Amir proposed SLAF (Simultaneous Learning and Filtering) in Learning Partially Observable Deterministic Action Models, demonstrating that exact belief tracking and action-model learning can be achieved concurrently without full state visibility.
Leveraging Hindsight to Train Partially Observable Agents
One of the hardest parts of learning under partial observability is credit assignment. If an agent takes an action and receives a penalty ten steps later, was it because of a bad choice, or because the agent misjudged the initial hidden state?
A powerful insight in recent reinforcement learning is that while the agent must act under partial observability during execution, we often have access to more information during training. We can use hindsight to resolve historical ambiguity.
Consider the framework proposed by Nota et al. in Posterior Value Functions: Hindsight Baselines for Policy Gradient Methods. They utilize information gathered after an action is taken to infer what the hidden state must have been at the time of the action. By evaluating past actions using this posterior knowledge, they drastically reduce gradient variance without introducing bias.
Similarly, Venuto et al. developed Policy Gradients Incorporating the Future (PGIF) in Policy Gradients Incorporating the Future, using a backward-running neural network to pull information from completed trajectories into the training phase. In the realm of theoretical guarantees, Zhang proved in Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation that leveraging these post-hoc revelations allows risk-sensitive agents to achieve polynomial regret bounds, matching the efficiency of much simpler, fully observable setups.
Neurosymbolic and Language-Driven World Models
With the advent of Large Language Models (LLMs), a new paradigm has emerged for handling partial observability in complex, text-based or embodied environments. Instead of maintaining purely statistical distributions over states, agents can use language and symbolic logic to "hypothesize" about the unobserved world.
In text-based environments, an agent rarely knows the layout of a house or the location of items initially. To bridge this gap, Jansen et al. developed PDDLEGO in PDDLEGO: Iterative Planning in Textual Environments. PDDLEGO uses an LLM to iteratively construct a formal, symbolic world model (expressed in PDDL) as it explores. The LLM acts as a flexible translator of partial observations, while a classical symbolic planner guarantees logical consistency.
This concept of using LLMs as dynamic belief generators is mirrored in Dynamic Planning with a LLM by Dagan et al.. Here, the language model generates plausible hypotheses about hidden environmental states using its common-sense priors (e.g., "if the milk is not on the counter, it is likely in the fridge"). These sampled beliefs fill the gaps in the world model, enabling classical planners to operate effectively despite incomplete data. Furthermore, Guan et al. in Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning demonstrate how refining these LLM-generated world models through environmental feedback allows agents to transition smoothly from highly uncertain initial states to successful task execution.
Active Information Gathering and Strategic Communication
An intelligent agent shouldn't just passively manage uncertainty; it should actively take actions to reduce it. This is known as active information gathering or active perception.
In human-robot collaboration, a robot must balance the urge to immediately assist with the need to explore hidden areas to confirm what the human actually needs. Puig et al. address this in NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants, showing how online probabilistic assistance allows a robot to calculate when to explore an unobserved cupboard versus when to hand a plate to a user.
When internal exploration isn't enough, agents can strategically communicate to resolve ambiguity. Kolb et al. demonstrated in Learning to request guidance in emergent language that agents can learn to recognize their own internal uncertainty. By equipping an agent with a "request bit," it learns to actively ask a partner for guidance precisely when its internal policy is insufficient due to partial observability, optimizing communication efficiency and task success.