In the idealized world of Reinforcement Learning (RL), time is a sequence of discrete, instantaneous turns. An agent observes a state $s_t$, chooses an action $a_t$, and the environment immediately transitions to $s_{t+1}$. This is the Markov property: the future depends only on the present.
In the real world, this assumption collapses. Sensors have latency, actuators have mechanical lag, and neural networks take time to compute a forward pass. By the time an agent receives an observation, the environment has already moved on. By the time its command reaches the motors, the situation has changed again. These delays introduce "memory" into the system, making it non-Markovian and causing standard RL algorithms to fail.
To fix this, researchers have converged on a powerful, intuitive solution: State Augmentation. By expanding the agent's input to include a history of past observations and "in-flight" actions, we can restore the Markov property and allow standard RL tools to function in delayed environments.
The Information Gap: Why Delays Break RL
When a system has a delay of $k$ time steps, the observation $o_t$ the agent receives is actually the state of the world at $s_{t-k}$. If the agent ignores this, it suffers from state aliasing—it treats a stale observation as the current truth.
As Dongsheng Li and William Wei Wang explain in Addressing Signal Delay in Deep Reinforcement Learning, the current true state is a function of that old observation plus every action the agent has taken since then. Without knowing those intervening actions, the agent cannot possibly know where the environment is "now."
Restoring the Markov Property via Action Buffering
The most common mechanism for fixing this is the Action Buffer. If there is a known delay, we augment the state vector with a history of recently issued but not yet realized actions.
In Delay-Aware Multi-Agent Reinforcement Learning, Baiming Chen and Ding Zhao formalize this as a Delay-Aware Markov Game (DA-MG). By including "in-flight" actions in the state, the transition once again becomes predictable from the current (augmented) input. This simple change allows algorithms like DDPG to work on robots with significant communication lags.
This isn't just a theoretical trick; it is a practical necessity for real-world hardware. For example:
- Low-cost Robotics: Rinu Boney and Arno Solin used observation and action stacking to enable the RealAnt quadruped to learn to walk in the real world, compensating for sensor noise and 500ms latencies.
- Industrial Control: In NFQ2.0: The CartPole Benchmark Revisited, Martin A. Riedmiller demonstrates that data stacking is often more critical for success on physical hardware than the choice of the RL algorithm itself.
- Space Robotics: Xueqian Wang and colleagues found that action-buffered state augmentation was more effective for controlling space manipulators than trying to build complex forward models to predict the current state.
The "Thinking Time" Problem
A subtle but pervasive form of delay is computation time. In standard RL, the environment pauses while the agent computes an action. In real-time systems, the environment keeps moving.
Simon Ramstedt and Christopher Pal addressed this in Real-Time Reinforcement Learning. They argue that the action currently being executed is actually a part of the state. By augmenting the observation with the ongoing action, the agent can reason about what the world will look like by the time its next action is finally ready. This turns a one-step computation lag into a Markovian framework.
Handling Stochastic and Variable Delays
While fixed delays are easier to model, real-world networks (like 5G or satellite links) have jitter. Harshad Khadilkar and Mayank Baranwal showed in Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays that we can handle this by using a fixed-size buffer and padding it with neutral tokens when delays are shorter than the maximum.
For even more complex scenarios, Bo Xia and Yilun Kong introduced DEER, which uses a pretrained encoder to map variable-length sequences of delayed observations into a fixed-length "delay-free" latent space. This allows a single policy to remain robust even as network conditions fluctuate.
Beyond Robotics: Temporal Logic and Biology
The principle of state augmentation extends beyond physical lag to any system with "memory":
- Temporal Tasks: When a task depends on a sequence of events (e.g., "Visit A, then B"), Austin Jones and Calin Belta show that augmenting the state with a history of visited regions allows RL to satisfy complex Signal Temporal Logic (STL) specifications.
- Biological Systems: In Reinforcement Learning for Control of Non-Markovian Cellular Population Dynamics, Josiah C. Kratz uses temporal frame-stacking to help RL agents discover drug-pulsing schedules for cancer cells that "remember" past exposures.
Summary of Intuition
If you are driving a car and close your eyes for one second, you cannot safely decide your next move based only on what you see when you open them. You must combine that new image with your memory of how fast you were going and how much you turned the wheel while your eyes were closed.
State augmentation is the mathematical equivalent of that memory. By appending the recent history of actions and observations to the current input, we provide the agent with the context it needs to bridge the gap between the stale past and the unobserved present.