Abstract
This paper addresses recursive markerless estimation of a robot's end-effector using visual observations from its cameras. The problem is formulated into the Bayesian framework and addressed using Sequential Monte Carlo (SMC) filtering. We use a 3D rendering engine and Computer Aided Design (CAD) schematics of the robot to virtually create images from the robot's camera viewpoints. These images are then used to extract information and estimate the pose of the end-effector. To this aim, we developed a particle filter for estimating the position and orientation of the robot's end-effector using the Histogram of Oriented Gradient (HOG) descriptors to capture robust characteristic features of shapes in both cameras and rendered images. We implemented the algorithm on the iCub humanoid robot and employed it in a closed-loop reaching scenario. We demonstrate that the tracking is robust to clutter, allows compensating for errors in the robot kinematics and servoing the arm in closed loop using vision.
Introduction
For a humanoid robot to perform precise manipulation tasks, such as grasping a cup or pressing a button, it must accurately determine the position and orientation (6D pose) of its end-effector. Traditionally, robots rely on internal joint encoders (proprioception). However, in complex humanoid platforms like the iCub, mechanical play, friction, and tendon elasticity often lead to significant discrepancies between the robot's internal state and its actual physical pose.
While computer vision can bridge this gap, markerless tracking of a robotic hand is challenging due to self-occlusion and environmental clutter. This research introduces a robust, markerless tracking system that utilizes the robot's own CAD models and real-time GPU rendering to achieve sub-pixel tracking accuracy in complex scenes.
Methodology
The core of the proposed system is a 3D model-aided particle filter (Sequential Monte Carlo filtering). The system maintains a distribution of hypothetical poses (particles). For each particle, the robot uses its internal CAD models and an OpenGL rendering engine to generate a virtual image of the hand as it would appear from the camera's perspective.
To compare these virtual images with real camera data, the system employs Histogram of Oriented Gradient (HOG) descriptors. HOG features are particularly effective at capturing structural shapes and edges while remaining invariant to lighting changes or minor color shifts. Particles that produce virtual images closely matching the HOG features of the real camera feed are assigned higher weights. To ensure real-time performance, the entire rendering and feature extraction pipeline is accelerated using NVIDIA CUDA.
Experimental Results
The system was validated on the iCub platform through two primary tasks. First, a reaching task tested accuracy across various starting positions and speeds. The system achieved a Mean Square Error (MSE) of 0.88 to 0.97 pixels, demonstrating precision that exceeds the requirements of most manipulation tasks.
Second, a "clutter test" required the robot to reach for a target (a ladybug plush) on a table crowded with diverse toys and household objects.
Despite the visual complexity, the robot achieved a 100% success rate over 10 trials. The HOG descriptors proved significantly more robust than traditional edge-matching techniques (like Chamfer distance), which often fail when background edges confuse the algorithm. Furthermore, the system successfully compensated for kinematic errors, enabling smooth closed-loop visual servoing.
Conclusion and Future Work
This work demonstrates that combining internal 3D models with robust visual descriptors allows humanoid robots to track their own limbs with high precision without external markers. Currently, the system focuses solely on the robot's hand and simplifies the model by ignoring underactuated fingers (the ring and little fingers) to maintain stability. Future research will explore joint estimation—tracking both the hand and the target object simultaneously—and refining kinematic models to include all finger segments for more complex grasping maneuvers.
Summary updated