TL;DR
This paper introduces a novel hybrid visual servoing system that combines multiple fixed cameras (Eye-to-Hand, EtoH) with an arm-mounted stereo camera (Eye-in-Hand, EinH) to precisely guide a robot arm to dynamic targets. By intelligently switching between these sensor modalities, the system overcomes challenges like partial occlusions and varying target distances, achieving more robust and accurate robotic manipulation.
What Problem Does This Paper Address?
Robots are increasingly being used in applications that require close interaction with humans, such as assisting with daily tasks. These applications often involve dynamic targets – things that move, like a person's hand or a cup being brought to their mouth. Guiding a robot arm to interact with such targets in real-time is challenging. Traditional visual servoing methods, which use camera feedback to control robot movement, face difficulties when:
- Multiple visual sources are combined: Integrating data from different cameras (fixed vs. arm-mounted) can be complex.
- Partial occlusions occur: When the target is hidden from view by the robot itself or other objects.
- Target distance varies significantly: Different sensors are optimal at different ranges.
This paper aims to address these challenges by developing a unified system that can adapt its visual sensing strategy to the current situation, ensuring accurate and reliable control of the robot arm for dynamic interaction tasks.
Methods
The core of the proposed approach is a hybrid multi-camera visual servoing system that intelligently switches between two main sensing configurations:
- Eye-to-Hand (EtoH): Utilizes four fixed RGBD (depth-sensing) cameras strategically placed around the workspace. This provides a global view and is effective for tracking targets at a distance or when the arm-mounted camera is occluded.
- Eye-in-Hand (EinH): Employs a stereo camera mounted on the robot arm's end-effector. This sensor offers high precision for close-range tasks.
A Master process acts as the central decision-maker. It monitors the target's distance and visibility. If the target is far or occluded from the EinH sensor, the Master activates the EtoH sensors. As the robot approaches, and if the target becomes visible and within range of the EinH sensor, the Master switches to the EinH sensor for fine-grained control. This switching mechanism includes a hysteresis threshold to prevent rapid oscillations between sensors.
For object tracking, the system uses color segmentation in the Lab color space followed by morphological post-processing to find the target's 2D image position. This is then combined with depth information from the selected sensors to estimate the target's 3D position. The EtoH system fuses data from the two closest and best-positioned Kinect sensors for improved accuracy.
The motion control is handled by a position-based controller that plans and executes trajectories in parallel. This allows for continuous operation and adaptation to the dynamic target. The system uses a Cartesian path planner to minimize a cost function that considers end-effector position, orientation, and joint configuration. A "discount" factor is incorporated to account for the unpredictability of target motion.
Datasets
The paper does not explicitly mention training datasets. Instead, it focuses on the experimental evaluation of the proposed system using a custom-built workcell. The experiments involve tracking and interacting with various targets:
- A red ball
- A bulls-eye target
- A smiley face (representing a person's mouth)
- A moving hand
These targets were used in four distinct experimental scenarios to demonstrate the system's capabilities.
Key Insights and Ideas
- Hybrid sensing is crucial for dynamic tasks: Combining the global view of EtoH sensors with the precision of EinH sensors offers a significant advantage over using either alone, especially when dealing with moving targets and occlusions.
- Intelligent sensor switching is key: A master controller that dynamically selects the best sensor based on target distance and visibility is essential for robust performance. This avoids relying on a single sensor that might be suboptimal in certain situations.
- EinH excels at fine-grained control: The arm-mounted stereo sensor provides superior accuracy for the final approach to the target, outperforming EtoH sensors when the target is within its operational range.
- EtoH compensates for EinH limitations: When the EinH sensor loses track of the target (due to occlusion or moving out of view), the EtoH system can re-acquire the target and guide the robot back into the EinH sensor's effective range.
- Parallel planning and execution is vital for real-time interaction: This approach allows the robot to continuously adapt its movements to a moving target without significant delays, improving efficiency and success rates.
Key Results
The paper demonstrates good performance across several challenging scenarios:
- Tracking Accuracy: The EinH sensor achieved a median error distance of 18 mm for a bulls-eye target, significantly better than the EtoH system's 38 mm error for a moving red ball.
- Ball Touching: The hybrid mode (EinH + EtoH) achieved a 95% success rate, compared to 68% for Kinect-only (EtoH). The hybrid approach also reduced the time to goal and the number of iterations.
- Bulls-eye Aiming: The EinH sensor improved accuracy in the final approach, with the hybrid system achieving 15 mm accuracy compared to 25 mm for Kinect-only.
- Delivery Tasks: The hybrid system achieved a 75% success rate for delivering an item to a moving hand, compared to 58% for Kinect-only. For the head and straw docking task, the system demonstrated its ability to insert a straw into a target mouth.
Limitations and Future Work
- Calibration Complexity: The authors acknowledge that the hybrid system's performance relies heavily on accurate calibration of multiple sensors, which can be challenging to implement.
- EinH Depth Resolution: The limited depth resolution of the eye-in-hand sensor (~7 mm) can affect targeting accuracy.
- Controller Limitations: The current position-based controller limits the servoing cycle to approximately 1 Hz, requiring the arm to stop before executing a new plan.
- Future Work: The authors plan to implement a velocity-based controller for the final approach to enable continuous operation and further improve performance. They also aim to address the calibration challenges and explore more advanced target tracking methods.
Summary updated