TL;DR
This paper introduces a novel teleoperation system that uses two independent mobile manipulators to extend a human operator's reach and manipulation capabilities into large or distant spaces. By combining IMU-based motion capture gloves for arm control and joysticks for base movement, the system allows for intuitive, dual-arm manipulation, effectively acting as a "body extension" for the human operator.
What Problem Does This Paper Address?
Teleoperation systems have long been used to perform tasks in environments that are dangerous, difficult, or impossible for humans to access directly, such as disaster sites or outer space. However, many existing systems are limited by a constrained workspace, often due to fixed bases or a single manipulator. Furthermore, while dual-arm systems offer advantages for complex manipulation, achieving coordinated control across independent mobile bases presents significant challenges. This paper addresses the need for a teleoperation system that not only expands the operator's reach into a larger workspace but also preserves the benefits of having two dexterous arms for cooperative manipulation, all while mitigating the complexities of controlling independent mobile platforms.
Methods
The core of the proposed system lies in its intuitive control interface and the architecture of the robotic setup.
- Human Interface: The system utilizes readily available commercial components:
- IMU-based Motion Capture Gloves (Perception Neuron): These gloves capture the precise 3D position and orientation of the human operator's hands. This data is directly mapped to control the poses of the robot arms' end-effectors (grippers) in task space.
- Joysticks (Nintendo Wii Nunchuk): These are used to control the movement of the mobile bases and the actions of the grippers (open/close).
- Robotic System: Two independent mobile manipulators are employed. Each manipulator consists of:
- Mobile Base: A non-holonomic base that allows for independent movement in a large area.
- 7-DoF Collaborative Robot Arm (UFACTORY xArm 7): Provides high dexterity for manipulation.
- Gripper: The end-effector for grasping objects.
- Depth Sensor: Integrated for visual sensing and feedback.
- Central Local Controller: Manages communication and coordination between the mobile base and the robot arm.
- Control Algorithm: An improved Levenberg-Marquardt (LM) method is used for solving the robot's inverse kinematics. This iterative approach continuously updates the robot's joint angles to follow the tracked human hand poses, ensuring smooth and precise motion. The method incorporates a damping term to avoid singularity and a clamping matrix to prevent joint angles from exceeding their effective ranges.
- Addressing Inherent Issues:
- Exceptional Tele-operation Motion: The system detects when a human operator performs unusually large or rapid hand movements (indicating an "exceptional motion"). In such cases, the robot temporarily stops following the trajectory to prevent dangerous movements or collisions. The system then monitors for the operator to "recatch" the robot, recalibrating if the positional and rotational errors are within acceptable limits.
- Calibration Before Dual-Arm Collaboration: To overcome the challenge of mapping human hand movements to two independently moving robot arms, a calibration step is crucial. The human operator can repeatedly adjust their pose and the robot configurations to achieve a well-coordinated state before performing collaborative tasks. This calibration is integrated into the exception handling routine, allowing operators to reset and recalibrate as needed.
Datasets
Not discussed in the paper. The experiments were conducted using the developed system with human operators performing various manipulation tasks.
Key Insights and Ideas
- Dual Mobile Manipulators for Expanded Workspace: The most significant insight is how using two independently mobile manipulators dramatically enlarges the effective workspace compared to single-base or fixed-base systems. This allows for manipulation of larger or longer objects that would be impossible otherwise.
- Preserving Dual-Arm Dexterity: The system successfully retains the advantages of human bimanual manipulation (coordination, dexterity) while extending the operator's reach, effectively creating a "body extension."
- Intuitive Control with Off-the-Shelf Components: The use of IMU gloves and standard joysticks makes the control interface accessible and user-friendly, demonstrating that advanced teleoperation doesn't necessarily require highly specialized hardware.
- Robustness through Exception Handling: The proactive detection and handling of "exceptional motions" are critical for safety and usability, preventing the robot from acting erratically based on unintended human movements.
- Importance of Calibration for Dual-Arm Coordination: The paper highlights that while independent mobility is key, careful calibration is essential for seamless dual-arm collaboration, especially when dealing with potential collisions or awkward operator poses.
Key Results
- Improved Inverse Kinematics: The iterative LM method demonstrated significantly lower position errors (norm of 0.00107) compared to using the robot's built-in IK solver without the iterative approach (norm of 0.0911).
- Successful Task Performance: The system was successfully used to perform various manipulation tasks, including:
- Single-arm pick-up (e.g., coffee bottle).
- Dual-arm handover of objects.
- Dual-arm pick and place of long and short sticks.
- Qualitative Findings: Experiments showed that the robot's motion accurately followed the human operator's hand trajectories, regardless of the operator's facing direction, due to the iterative nature of the LM method. While tasks were successfully completed, operators sometimes required multiple corrections, indicating a learning curve and the importance of calibration. Task completion times varied, with dual-arm tasks generally taking longer (e.g., 120s for handover).
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Calibration Burden: The current reliance on manual calibration for dual-arm coordination is a significant limitation. Future work should focus on developing methods that require less calibration effort.
- Joystick Interface: The current joystick design requires the operator to hold it horizontally, which is not always intuitive. Exploring joysticks with embedded IMUs could improve ergonomics.
- Limited Coordination Control: The system currently struggles with highly coordinated manipulation of objects due to potential errors between the two arms. A higher-level synchronized control mode, where the operator controls the object and the robots generate coordinated motion, is proposed.
- Lack of Feedback for Unseen Workspaces: The current system lacks feedback mechanisms for teleoperation in distant or visually occluded environments. Developing such feedback is a key area for future development.
- Collision Avoidance: While the LM method helps prevent out-of-range joint failures, explicit collision avoidance with environmental obstacles is not fully integrated and remains an open problem.
Summary updated