Abstract
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving".
Introduction
Standard Deep Reinforcement Learning (RL) typically operates under a "blocking" paradigm: the agent observes the environment's state, the world effectively pauses while the agent computes an action, and then that action is applied. While this abstraction works well for turn-based games or slow-speed simulations, it breaks down in real-world robotics. In physical systems, the time required to process sensory data and compute a policy output—latency—is non-negligible. By the time a robot initiates a movement, the environment has already evolved beyond the state it initially observed.
To address this, researchers often resort to slowing down the robot or utilizing specialized low-latency hardware. This paper proposes a more fundamental solution: reformulating the underlying mathematics of RL to explicitly account for concurrent control, where the agent "thinks" about its next move while the previous one is still in progress.
Methods: Concurrent Control Framework
The core of the proposed approach is a framework that treats the agent and the environment as parallel processes. The authors derive a modified version of the Bellman equations for both continuous and discrete time that incorporates system delays.
As shown in the figure above, the framework explicitly models the "spillover" time ($t_{AS}$), which is the duration during which the previous action continues to execute while the next action is being selected. To enable effective learning in this setting, the agent's Q-function is augmented with three key features:
- Previous Action ($a_{t-1}$): The action currently being executed.
- Action Selection Time ($t_{AS}$): The measured latency of the computation.
- Vector-to-go (VTG): A novel feature representing the remaining portion of the current action yet to be completed at the moment of observation.
These features are integrated into standard value-based DRL algorithms, such as DQN or QT-Opt, with minimal architectural changes.
Experimental Evaluation
The methodology was tested across three distinct domains:
- Toy Control Tasks: 3D MuJoCo-based versions of Cartpole and Pendulum, modified to simulate varying levels of system latency.
- Large-Scale Simulated Grasping: A 7-DoF robotic arm tasked with picking objects from a bin in a simulated environment.
- Real-World Robotic Grasping: A physical 7-DoF robot arm performing grasping tasks using an over-the-shoulder camera.
Results and Discussion
The experiments revealed several key insights. Most notably, the Vector-to-go (VTG) feature proved to be the most critical component for performance. By providing the agent with a precise understanding of its current trajectory and momentum, VTG allows for seamless transitions between actions, resulting in smoother, more human-like motions.
As illustrated in the ablation studies for Cartpole and Pendulum, models utilizing VTG significantly outperformed unconditioned baselines and those using only basic previous action information.
In real-world robotic grasping, the concurrent agent achieved a 49% increase in execution speed compared to the blocking baseline. In simulation, the approach reduced total wall-clock time per episode by 31.3% while maintaining or improving grasp success rates (93.49% in simulation, 68.6% in real-world tests). This demonstrates that concurrent control can significantly enhance efficiency without sacrificing task accuracy.
Conclusion and Future Work
This work establishes a theoretically sound and practically effective framework for "thinking while moving" in RL. By modeling system delays directly within the Bellman equations, robots can achieve higher speeds and smoother control.
Future research directions include adapting the framework for systems with highly variable sampling periods and extending the concurrent logic to on-policy and policy gradient methods. Additionally, testing the framework in highly dynamic environments remains an exciting frontier.
Summary updated