The foundational assumption of standard Reinforcement Learning (RL) is the Markov property: the agent's current observation provides a complete and sufficient description of the environment's state. Under this assumption, an optimal action can be chosen based solely on the immediate present.
In the real world, however, the Markov property routinely breaks down. A single camera frame cannot reveal the velocity of a moving vehicle; a haptic sensor cannot instantly detect the mass of an object; a network node cannot see global traffic congestion from an isolated packet drop. These scenarios are Partially Observable Markov Decision Processes (POMDPs) or non-Markovian environments.
To solve them, agents must look backward to move forward. Integrating recurrent neural network layers (such as LSTMs or GRUs) directly into RL architectures allows agents to maintain an internal, evolving hidden state. This recurrent state acts as a latent state estimator, compressing a history of past observations and actions into a sufficient statistic for decision-making.
Restoring the Markov Property via Memory
When an environment is partially observable, the optimal policy depends not just on the current observation $o_t$, but on the entire history of interactions $H_t = (o_1, a_1, \dots, o_t)$. Treating $H_t$ as a raw input is computationally intractable because its length grows over time.
Recurrent reinforcement learning solves this by introducing a recurrence relation:
$$h_t = f(h_{t-1}, o_t, a_{t-1})$$
where $h_t$ is the internal hidden state. As Schmidhuber early on demonstrated, this mechanism allows networks to internalize temporal dependencies, effectively mapping non-Markovian control tasks into a space where hidden environmental variables can be inferred.
By mapping the transient, incomplete signals of the environment into a persistent internal state, the agent transforms a POMDP back into a standard decision process, as formalized by Rombouts et al.. Rather than trying to predict the future from an isolated snapshot, the policy operates over $h_t$, which serves as a rich, low-dimensional summary of the past (Gao et al.).
[Environment] ---> o_t ---------------> [ Recurrent Layer ] ---> h_t ---> [ Policy ] ---> a_t
^ |
| v
+--- (Previous Memory) ---+
To make this hidden state truly useful, the agent must bootstrap its understanding of time. Makino showed that combining temporal-difference learning objectives with recurrent architectures allows agents to ground their internal representations in predictive targets, making the learning of complex temporal dependencies significantly more efficient than using standard recurrent networks alone.
Inferring Hidden Physical Properties
One of the most powerful aspects of a recurrent state representation is its ability to act as an uncalibrated sensor. By observing how an environment responds to actions over multiple time steps, the recurrent state can infer latent physical properties that are completely invisible in a single frame.
Consider a robot attempting to manipulate an object or traverse terrain without vision:
- Terrain Estimation: Siekmann et al. demonstrated that a blind bipedal robot can reliably climb stairs by using recurrent layers to process sequences of joint positions and velocities. The network infers hidden environmental states, such as step height, purely from the temporal pattern of its proprioceptive feedback.
- System Dynamics: In robotic assembly, Nguyen et al. utilized recurrent states to integrate haptic and proprioceptive histories, resolving ambiguities during contact tasks under partial observability.
- Unobservable Parameters: Similarly, Chiappa et al. showed that a recurrent policy can decode unobservable environmental properties like object mass and friction coefficients from sequences of sensory inputs, acting as an implicit system identifier.
Tracking Derivatives and Trends
In many applications, the absolute value of an observation matters less than its rate of change. Memory allows an agent to compute implicit derivatives—such as velocity, acceleration, or degradation slopes—from a sequence of static inputs.
In autonomous driving, extracting these derivatives is a prerequisite for safety. Krishna et al. and Chen et al. demonstrated that recurrent layers allow agents to perceive vital temporal dynamics, such as acceleration and relative velocity, directly from sequences of static images or noisy sensor snapshots. This temporal context prevents performance collapse when sensors are noisy or occluded (Naveed et al.).
This capacity extends far beyond physics. In strategic decision-making, such as Formula One race strategy, the authors of Explainable Reinforcement Learning for Formula One Race Strategy showed that recurrent layers are highly effective because they capture resource degradation slopes and closing speeds over time, rather than relying on static snapshots of tire wear or track position.
The same principle applies to network optimization. Wydmański et al. used recurrent layers to analyze historical sequences of network observations, allowing an agent to distinguish between transient channel noise and sustained trends in network congestion.
Bridging Occlusions and Signal Loss
When an environment is dynamic, objects frequently move out of sight. Without memory, an object that is occluded ceases to exist for the agent, leading to catastrophic failures or erratic behavior. Recurrent state representations maintain a persistent belief state that bridges these temporal gaps.
Time t-1: Target Visible ---> Recurrent state stores location.
Time t: Target Occluded ---> Recurrent state updates location via velocity tracking.
Time t+1: Target Re-emerges ---> Agent seamlessly continues tracking.
This mechanism is vital across several domains:
- Sensor Field-of-View: Chen showed that an agent with a narrow field-of-view camera can avoid 3D obstacles by using a recurrent network to "remember" obstacles that have passed out of its immediate line of sight.
- Extreme Environments: For quadrupedal robots navigating in the wild, Miki et al. used an attention-based recurrent encoder to preserve a persistent belief state of the terrain, ensuring the robot can safely step onto previously perceived ground even when its vision sensors are temporarily blinded by snow, fog, or reflections.
- Intermittent Signal Loss: In underwater target tracking, O’Reilly et al. demonstrated that passing the hidden state across time steps allows an actor-critic framework to maintain tracking performance through periods of severe data sparsity and acoustic signal dropouts.
Path-Dependency and Endogenous Dynamics
Many complex systems exhibit path-dependency, where the effect of an action depends heavily on the cumulative history of what came before. Recurrent states excel here because they can track slow, cumulative environmental shifts.
For example, Tang et al. integrated recurrent networks into a multi-agent proximal policy optimization framework to manage the gradual, cumulative wear and tear on hardware components across multiple time steps. In wireless networks, Chu et al. utilized LSTMs to capture the long-term temporal dependencies inherent in energy harvesting patterns and battery state evolution, allowing for sustainable long-term resource allocation.
Remarkably, when recurrent agents are embedded in environments with strong temporal patterns, they can develop internal, self-sustaining dynamics. Labash et al. discovered that training an agent with recurrent memory in a periodic day-night cycle causes the spontaneous emergence of internal periodic orbits within the latent state space. These endogenous dynamics act as a biological clock, allowing the agent to anticipate environmental shifts even when external cues are completely removed.
Ultimately, whether an agent is navigating a mapless maze (Seong et al.), tracking blood vessels in robotic ultrasound navigation (Bi et al.), or modeling fluid dynamics (Terpin), recurrent state estimation serves the same fundamental purpose: it turns time into a feature, providing the stable context required to act optimally in an uncertain, changing world.