Abstract
This article presents a novel telemanipulation system to advance aerial manipulation in dynamic and unstructured environments. The proposed system features not only a haptic device, but also a virtual reality (VR) interface that provides real-time 3-D displays of the robot’s workspace, as well as a haptic guidance to its remotely located operator. To realize this, multiple sensors, namely, a LiDAR, cameras, and inertial measurement units (IMUs) are utilized. For processing the acquired sensory data, pose estimation pipelines are devised for industrial objects of known and unknown geometries. We further propose an active learning (AL) pipeline in order to increase the sample efficiency of a pipeline component that relies on deep neural networks (DNNs)-based object detection. All of these algorithms jointly address various challenges encountered during the execution of perception tasks in industrial scenarios. In the experiments, exhaustive ablation studies are provided to validate the proposed pipelines. Methodologically, these results commonly suggest how an awareness of the algorithms’ own failures and uncertainty (“introspection”) can be used to tackle the encountered problems. In addition, outdoor experiments are conducted to evaluate the effectiveness of the overall system in enhancing aerial manipulation capabilities. In particular, with flight campaigns over days and nights, from spring to winter, and with different users and locations, we demonstrate more than 70 robust executions of pick-and-place, force application, and peg-in-hole tasks with the DLR cable-suspended aerial manipulator (SAM). As a result, we show the viability of the proposed system in future industrial applications.
Introduction
Maintaining critical infrastructure like wind turbines or oil refineries is dangerous and expensive, often requiring human technicians to work at extreme heights. While aerial robots—drones equipped with robotic arms—offer a potential solution, they face a "triple threat" of technical challenges: instability due to the "floating base" of a flying platform, perception failures in changing outdoor environments, and high operator cognitive load when controlling via traditional 2D camera feeds.
This research aims to bridge the gap between laboratory success and real-world industrial utility by creating a system that is "self-aware" enough to know when its sensors are failing, while providing a high-fidelity interface for human operators.
System Architecture and Methodology
The researchers utilized the DLR cable-Suspended Aerial Manipulator (SAM). The system architecture is built upon a closed-loop teleoperation framework that integrates force feedback and immersive visualization.
Introspective VR Creation
Rather than streaming high-bandwidth 3D point clouds over potentially unstable wireless connections, the robot generates a "Scene Graph." It estimates the 6D pose of objects and transmits only these coordinates to the operator's VR headset. The system's "introspection" is critical: it monitors its own uncertainty. If a visual marker is lost or shadowed, the system automatically fuses data from visual-inertial SLAM and IMUs to maintain object tracking until the primary sensors recover.
Perception for Unknown Objects
For objects without pre-existing CAD models, such as irregular industrial piping, the system employs a LiDAR-based pipeline. This combines LiDAR Odometry and Mapping (LOAM) with deep learning to detect and reconstruct object geometry in real-time.
Active Learning (AL) for Robustness
To ensure Deep Neural Networks (DNNs) remain robust in varying outdoor conditions, the authors implemented Bayesian Neural Networks. These networks provide an uncertainty score alongside their predictions. The system uses this score to identify the most "informative" images for human labeling. This Active Learning approach allowed the system to reach 95% of its peak performance using only 25% of the available training data, facilitating rapid adaptation from sunny labs to snowy fields.
Experimental Results and Robustness
The system was validated through custom-collected field data, including 20,000 images across diverse scenes and real-world flight data from over 70 task executions.
Key performance metrics include:
- High Precision: Successful peg-in-hole tasks with an error margin of less than 2.5mm.
- Efficiency: The VR interface with haptic guidance (VR+HG) reduced task execution time by approximately 1.8x compared to traditional camera-only teleoperation.
- Environmental Resilience: The system demonstrated long-term robustness across seasons (spring to winter) and lighting conditions (day and night).
- Haptic Guidance: Providing "virtual fixtures" (haptic guidance toward targets) significantly lowered the difficulty of high-precision tasks for operators.
Conclusion
This work demonstrates that "introspection"—the ability of an algorithm to recognize its own failures—is a powerful safety net for aerial manipulation. By combining this self-awareness with immersive VR and haptic feedback, the system enables complex industrial tasks to be performed remotely with high precision. Future work will focus on larger-scale user studies and moving the Active Learning loop to a fully on-board, real-time process to further enhance adaptability in the field.
Summary updated