In classical reinforcement learning (RL), we often design controllers under the assumption of a Fully Observable Markov Decision Process (MDP). This framework assumes that the robot's current sensory snapshot contains all the information necessary to select an optimal action.
In the real world, this assumption quickly falls apart. A single frame from a camera or a single reading from a joint encoder cannot capture latent physical properties like surface friction, the exact mass of an object, sensor latencies, or the hidden intentions of moving obstacles. To handle these real-world complexities, modern control architectures are shifting from reactive, frame-by-frame mapping toward history-dependent policies. By integrating memory mechanisms—such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Linear Recurrent Units (LRUs)—these architectures allow agents to maintain an internal state that tracks information over time.
Implicit System Identification and Sim-to-Real Transfer
One of the most powerful reasons memory-augmented control works is its ability to perform implicit system identification. When deploying a robot from a simulator to the real world, engineers frequently use Domain Randomization (DR), varying physical parameters like mass, friction, and motor gains during training so the policy can tolerate real-world discrepancies.
A memoryless policy must find a single, conservative compromise that performs passably across all randomized environments. In contrast, a history-dependent policy can figure out exactly which environment it is currently operating in. As demonstrated by Xiaoyu Chen et al. in Understanding Domain Randomization for Sim-to-real Transfer, incorporating memory allows an agent to use historical observations to distinguish between different environmental dynamics. If a robot applies a specific torque and its joint moves slower than expected, the history contains the data needed to infer that the joint friction or payload mass is high.
[ Past Actions + Past Observations ] ---> [ Recurrent Memory Layer ] ---> [ Inferred Latent Dynamics ] ---> [ Optimized Action ]
This capacity to resolve model uncertainty online is further leveraged by Tianwei Ni et al. in Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism. The authors utilize history-dependent encoders, such as linear recurrent units, to maintain a running posterior distribution over an ensemble of world models. Rather than relying on strict, pessimistic constraints to remain safe, the agent uses the history of the current episode to implicitly identify the most accurate world model at test-time, adapting its actions dynamically to the specific transitions it encounters.
To extend this adaptability to entirely unseen tasks, Scott Jeen et al. introduce Foundation Policies with Memory. By feeding memory states into both the actor (action selection) and the critic (value estimation) components of a foundation policy, the authors enable generalist agents to adjust their decision-making on the fly when environmental dynamics shift.
Stabilizing Locomotion and Manipulation Under Uncertainty
Beyond identifying static physical parameters, temporal memory is vital for smoothing out high-frequency noise, sensory delays, and physical coordination challenges.
Robust Locomotion
In legged locomotion, a robot must react to sudden terrain changes and external forces without explicit external timing references. Aayam Kumar Shrestha et al. demonstrate this in Revisiting Reward Design and Evaluation for Robust Humanoid Standing and Walking, showing that a stateful recurrent policy trained via policy optimization allows humanoid robots to withstand significant physical disturbances without relying on pre-recorded human motions or rigid clock signals.
When sensory feedback is delayed—a common issue with onboard vision systems—memory bridges the gap. Guillaume Bellegarda et al. show in Visual CPG-RL: Learning Central Pattern Generators for Visually-Guided Quadruped Locomotion that memory-enabled recurrent networks like LSTMs significantly enhance locomotion robustness and energy efficiency compared to memory-free networks by retaining past information to counteract sensory lag.
This temporal context becomes even more critical when multiple robots must cooperate without explicit communication channels. In Learning Decentralized Multi-Biped Control for Payload Transport, Addison Johnson et al. deploy recurrent layers within a decentralized control policy. The memory allows individual bipedal robots to maintain a history of recent movements and force interactions, enabling them to implicitly infer the shifting center of mass of a shared payload and stabilize the system collaboratively.
Smooth Manipulation and Navigation
In robotic manipulation, workspaces are frequently occluded by the robot's own arms or changing camera angles. Alexander Herzog et al. demonstrate in Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators that recurrent architectures are essential for tracking the original state of a workspace during manipulation under partial observability.
Furthermore, memory directly impacts the quality of motion. Frame-by-frame vision controllers often produce erratic, jerky movements because they react aggressively to minor visual fluctuations. As shown by Weizi Li et al. in ADAPS: Autonomous Driving Via Principled Simulations, integrating memory-enabled architectures into vision-based control policies smooths out these trajectories, resulting in more stable navigation. This continuous temporal awareness is similarly utilized by Weihao Yuan et al. in Reinforcement Learning in Topology-based Representation for Human Body Movement with Whole Arm Manipulation, where recurrent units allow a controller to maintain a memory of previous states to react smoothly to continuous human motion during dynamic rescue operations.
Overcoming the Training Challenges of Recurrent Policies
If history-dependent policies are so effective, why aren't they used universally? The primary bottleneck is that training recurrent neural networks within reinforcement learning loops is notoriously difficult. Backpropagation Through Time (BPTT) suffers from vanishing and exploding gradients, and standard recurrent RL algorithms are often sample-inefficient and unstable.
To bypass the difficulties of training standard recurrent architectures, Marvin Mengxin Zhang et al. propose an alternative approach in Policy Learning with Continuous Memory States for Partially Observed Robotic Control. Instead of relying on standard hidden states, the authors introduce continuous memory variables that the policy explicitly reads from and writes to, optimizing them via Guided Policy Search (GPS). This strategy bypasses traditional RNN training hurdles, achieving high precision and task success in significantly fewer iterations.
[ Observation ] ---> [ Guided Policy Search ] ---> [ Explicit Read/Write ] <---> [ Continuous Memory State ]
When dealing with decentralized systems under severe communication constraints, managing historical context requires specialized optimization. Anam Tahir et al. address this in Collaborative Optimization of the Age of Information under Partial Observability by combining Proximal Policy Optimization (PPO) with recurrent network architectures. This combination allows decentralized sensors to maintain a temporal context of historical observations, optimizing data freshness even when network delays or missing feedback loops obscure the true state of the system.
Tracking Long-Horizon Context and Procedural States
Memory-augmented architectures do not just smooth out local physical control; they also allow agents to track procedural state across long horizons where the optimal next action depends entirely on what has already been accomplished.
This requirement is clearly visible in sequential image manipulation. Fatemeh Azimi et al. demonstrate in A Reinforcement Learning Approach for Sequential Spatial Transformer Networks that incorporating recurrent units into a policy network is critical for satisfying the Markov property in sequential image processing. Because the current state of an image alone does not reveal the sequence of transformations already applied, the agent must rely on memory to choose the next adjustment.
For long-horizon visual planning, standard sequence models can diverge over extended rollouts. To solve this, Boyuan Chen et al. introduce Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion, an approach that combines recurrent architectures with diffusion-based denoising. This framework allows robotic planning systems to maintain task-relevant information over long periods, providing robustness against visual distractors and preventing trajectory divergence.
In hierarchical control, memory allows high-level planners to guide low-level motor skills based on episode-specific context. As shown by Josh Merel et al. in Hierarchical Visuomotor Control of Humanoids, integrating recurrent memory into a high-level controller enables humanoid agents to navigate complex environments by remembering episode-specific rules or object properties, such as distinguishing between beneficial and harmful targets based on past observations.
Finally, this concept extends to text-based procedural tasks. In ScriptWorld: Text Based Environment for Learning Procedural Knowledge, Abhinav Joshi et al. demonstrate that recurrent policy optimization is necessary for navigating multi-step real-world chores. By integrating memory units into the RL architecture, the agent maintains a history of completed sub-tasks, ensuring it can successfully execute long-horizon sequences where the next correct action depends entirely on what steps were previously completed.
Key Foundational Work
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2018) — contributed 2 concept elements to this direction.
Reinforcement Learning in Topology-based Representation for Human Body Movement with Whole Arm Manipulation (2018) — contributed 1 concept elements to this direction.
Policy Learning with Continuous Memory States for Partially Observed Robotic Control (2015) — contributed 1 concept elements to this direction.