Abstract
While quadrupedal robots have achieved significant milestones in navigating complex terrains, high-speed dynamic interactions—such as catching a fast-moving object—remain a challenge due to the inherent latency of traditional vision systems. This paper presents a novel approach that combines neuromorphic sensing with reinforcement learning to enable a quadrupedal robot to catch objects traveling at speeds up to 15 m/s. By utilizing an event camera to minimize perceptual latency and a neural network policy for agile maneuvering, the system achieves an 83% success rate in real-world experiments. This work demonstrates the feasibility of using event-based vision for high-speed, onboard robotic perception and control.
Introduction
Quadrupedal platforms like the ANYmal or Spot have demonstrated remarkable stability and mobility in static or slowly changing environments. However, achieving "animal-like" agility in response to high-speed external stimuli remains a frontier in robotics. A primary obstacle is the latency-bandwidth tradeoff found in traditional frame-based RGB cameras. To track an object moving at 15 m/s (54 km/h), a standard camera requires a very high frame rate, which generates a massive volume of data that often exceeds the processing capabilities of onboard mobile hardware. Conversely, lower frame rates introduce significant perceptual latency, where the object's position has changed substantially by the time the frame is processed.
This research addresses these challenges by integrating an event camera—a sensor that reports pixel-level intensity changes with microsecond latency—with a robust reinforcement learning (RL) framework. The result is a system capable of real-time trajectory prediction and agile "lunging" maneuvers to catch objects with high precision.
Methodology
The proposed system architecture is divided into a modular pipeline consisting of a visual frontend for perception and a planning backend for motor control.
Visual Frontend: Event-based Perception
Unlike traditional cameras that capture full frames at fixed intervals, event cameras only transmit data when a pixel detects a change in brightness. This sparse data stream is ideal for high-speed tracking. The perception pipeline involves three key steps:
- Motion Compensation: To isolate the moving object from the background, the system uses Inertial Measurement Unit (IMU) data to subtract the robot's own ego-motion from the event stream.
- Object Detection: The remaining events, which represent the moving ball, are clustered using the DBSCAN algorithm.
- Trajectory Fitting: Operating with a monocular setup, the system estimates depth by assuming a known object size. It then applies RANSAC to fit a ballistic parabola to the detected 3D points, allowing the robot to predict the exact time and location of the ball's arrival.
Planning Backend: Reinforcement Learning Control
Traditional model-based controllers often struggle with the non-linear, high-acceleration maneuvers required to position a net for a catch. To overcome this, the researchers trained a neural network policy using Proximal Policy Optimization (PPO) within the Isaac Gym simulation environment.
The policy receives the predicted impact point and the robot's proprioceptive state (joint positions and velocities) as inputs. It then outputs target joint deviations. This allows the robot to perform complex tilting and side-stepping behaviors that prioritize the position of the mounted net over traditional gait stability, enabling the robot to "lunge" toward the target.
Experimental Setup and Results
Training and Validation
The RL policy was trained entirely in simulation, where thousands of throws were modeled to ensure robustness. Crucially, the policy was deployed on a physical ANYmal C robot using a "zero-shot" sim-to-real transfer, requiring no fine-tuning on the physical hardware. The system was evaluated through 22 real-world experiments with balls thrown at speeds ranging from 5 to 15 m/s.
Performance Metrics
- Success Rate: The robot achieved an overall success rate of 83% across all test speeds.
- Vision Reliability: The event-based vision algorithm correctly identified the impact point in over 93% of trials for objects traveling under 9 m/s.
- Computational Efficiency: The entire pipeline runs at 100 Hz on an NVIDIA Jetson Orin, demonstrating that the approach is suitable for resource-constrained mobile platforms.
- Latency Advantage: A formal "Worst-Case Latency Analysis" conducted by the authors proved that event cameras are mathematically superior for this task. Because they do not wait for a frame to complete, they can provide the minimum two points required for trajectory fitting much faster than frame-based alternatives.
Discussion and Limitations
The success of this system highlights the potential of neuromorphic sensing in robotics. The RL policy learned to utilize the robot's full range of motion, often adopting body tilts and lunges that would be difficult to program manually.
However, several limitations remain. The current monocular setup requires prior knowledge of the object's size to estimate depth; moving to a stereo event-camera configuration could generalize the system to unknown objects. Additionally, the robot's current lateral range is limited to approximately 0.6 meters from its center. While the robot can lunge effectively, it cannot yet combine high-speed locomotion with catching for objects thrown significantly wide of its position. Future work will focus on integrating more dynamic stepping strategies to expand this operational envelope.
Conclusion
This work represents the first successful demonstration of a quadrupedal robot catching high-speed objects using only onboard sensing. By breaking the latency cycle associated with traditional vision and leveraging the flexibility of reinforcement learning, the researchers have moved a step closer to achieving true animal-like agility in robotic systems.
Summary updated