Reactive Robotic Grasping of Moving Objects

48 concepts48 papers1 proposal

Imagine trying to grab a coffee cup off your desk. If the cup is stationary, your brain can take its time: it observes the cup's position, plans a trajectory for your arm, and executes the reach. This "see-plan-execute" pipeline is the foundation of traditional robotics.

But what if someone slides the cup across the table, or tosses a ball into your hand? If you rely on a slow, sequential planning loop, your hand will consistently arrive where the object used to be. To interact with a changing world, robots must abandon the assumption of a static environment and transition to real-time, reactive manipulation.

The Illusion of a Static World

Most existing robotic manipulation frameworks operate under a strict "quasi-static" assumption. As highlighted by Bousmalis et al. and Qin et al., grasping models typically assume that environments and objects remain entirely stationary until the exact moment of physical contact. This simplifies the mathematics of motion planning, allowing algorithms to treat a 3D scene as a fixed geometric puzzle.

However, Fang et al. point out that this limitation completely excludes dynamic real-world scenarios where objects shift, fall, or are moved by external forces during the manipulation process. If a robot is picking items from a fast-moving conveyor belt or participating in a human-to-robot handover, an open-loop plan will fail the moment the target deviates from its initial coordinates.

Breaking the Latency Barrier

Why can't we simply run traditional static grasping algorithms at a higher frame rate? The bottleneck is computational latency.

Modern high-capacity models, such as Vision-Language-Action (VLA) transformers, excel at understanding semantic context and generalizing to novel objects. Yet, as Zhang et al. observe, running these massive models for every single motor command introduces significant latency. This computational delay makes robots clumsy and unresponsive in fast-paced environments.

To bridge this gap, researchers split the problem into two parts: fast, asynchronous perception and predictive state estimation.

[High-Latency Perception] ---> [Predictive Filter (e.g., EKF)] ---> [High-Frequency Reactive Policy]
       (Tracks Object)              (Fills Perception Gaps)             (Adjusts Motor Commands)

Instead of waiting for a heavy visual model to process a full frame, event-based vision sensors capture pixel-level changes asynchronously. Barranco et al. demonstrated that redefining clustering algorithms to operate on these continuous event streams slashes computational costs while identifying "fast affordances"—the immediate interaction potential of an object—in real time.

When sensor data is delayed or temporarily occluded, the robot must predict the object's future trajectory. Boularias et al. addressed this by sandwiching an Extended Kalman Filter (EKF) between the perception system and a reinforcement learning policy. If the robot's camera loses sight of a moving target during the final approach phase, the EKF estimates its trajectory, allowing the control loop to maintain accuracy and recover from failed attempts.

Closed-Loop Reactivity and Trajectory Adaptation

To successfully intercept a moving target, a robot's controller must continuously map its current relative state to an immediate motor adjustment. This requires closed-loop control laws that prioritize stability and speed.

One approach is to embed classical control principles directly into deep learning architectures. Yu et al. introduced a framework that embeds Control Lyapunov Function principles into a neural network. This mathematical constraint ensures that the robotic arm's trajectory smoothly and stably converges on the target, enabling real-time reactive adjustments at a high frequency (85Hz) using only monocular vision.

Traditional Planning:  [Initial State] -----------------------------> [Fixed Target]
Reactive Planning:     [Current State] -> [Adjust] -> [Adjust] -> [Moving Target]

Another strategy is to make the policy anticipate the final grasp configuration while it is still moving. Wang et al. developed a closed-loop policy that utilizes a "goal-auxiliary" task. As the gripper approaches an object during a handover, the network simultaneously predicts the final 6D grasp pose and determines the immediate next motor action. This dual focus allows the robot to dynamically reshape its trajectory in response to unpredictable human movements.

Furthermore, tracking a moving object means its reachability changes dynamically. Marturi et al. resolved this by implementing a dual-planner system: a global planner rapidly re-positions the arm if the target moves into a completely different sector of the workspace, while a local planner handles smooth, high-frequency tracking for the final capture.

Spatiotemporal and Generative Representations

As manipulation moves from rigid blocks to dynamic environments, our representations of grasps must evolve from static 3D coordinates to spatiotemporal trajectories.

Instead of treating a grasp as a single snapshot in time, Ye et al. proposed the Continuous Grasping Function, which models multi-fingered hand trajectories as a continuous implicit function of time. This formulation allows a dexterous hand to generate smooth, human-like closing sequences that naturally adapt to external disturbances or moving targets.

For highly explosive, high-speed actions like catching a thrown object, generative models have proven highly effective. Zhao et al. combined human intent with geometry-aware diffusion policies to adaptively guide multi-fingered hands during 3D object catching.

When the entire robot is in motion—such as a quadrupedal robot equipped with a manipulation arm—the coordination problem scales significantly. Liang et al. tackled this by combining a teacher-student reinforcement learning framework with a specialized "grasp memory bank," enabling mobile robots to coordinate their legs, chassis, and arms simultaneously to snatch moving objects across uneven terrain.

By shifting from static geometric planning to predictive, high-frequency feedback loops, modern robotic manipulation is overcoming the latency barrier. The result is a generation of robots that do not just interact with a frozen world, but fluidly adapt to the dynamics of our own.

Key Foundational Work

Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping (2018) — contributed 1 concept elements to this direction.
Real-time clustering and multi-target tracking using event-based sensors (2018) — contributed 1 concept elements to this direction.
Statistical Learning by Imitation of Competing Constraints in Joint Space and Task Space (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_object_manipulation_0cf1bf81,
  title = {dynamic object manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-object-manipulation/rdr_0cf1bf81ac65465fb61e925515498379},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_object_manipulation_0cf1bf81_v1,
  title = {dynamic object manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-object-manipulation/rdr_0cf1bf81ac65465fb61e925515498379/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake