Introduction
Aerial manipulation—performing physical tasks like welding, repairs, or cleaning using Unmanned Aerial Vehicles (UAVs)—is a burgeoning field with significant technical hurdles. The primary challenges involve dynamic coupling, where the movement of a robotic arm shifts the UAV's center of mass and creates destabilizing counter-torques, and the complexity of navigating dynamic environments. This paper introduces a hybrid control framework that decouples flight navigation from arm control, combining Time-To-Collision (TTC) for obstacle avoidance with Q-learning for precise end-effector trajectory tracking.
Methodology: A Three-Stage Pipeline
The authors propose a modular three-stage pipeline designed to work alongside standard flight stacks like ArduPilot or PX4. This separation allows the system to handle complex manipulation tasks without requiring highly integrated, custom flight firmware.
- Kinematic Mapping: The system first scans the environment and defines the target trajectory for the end-effector. Using inverse kinematics, it determines a "kinematic feasible region"—the specific volume of space where the UAV can fly while keeping the target within the physical reach of the arm.
- TTC-Based Motion Planning: To navigate safely, the system uses a reactive Time-To-Collision (TTC) model. This approach estimates the time remaining before a potential collision based on velocity and distance, allowing the drone to plot a safe path within the previously defined feasible region.
- Reinforcement Learning (Q-Learning): A model-based Q-learning agent is trained to control the two-degree-of-freedom (2-DOF) arm. The agent receives the UAV's pose and current joint angles as state inputs and outputs the optimal joint torques to minimize the distance between the end-effector and the target trajectory.
The Kinematic Feasible Region
A key innovation of this research is the explicit modeling of the "feasible region." Rather than treating the drone and arm as a single complex entity for path planning, the system calculates a "tunnel" of space that respects the arm's mechanical limits (joint angles and link lengths).
Results and Performance
The proposed framework was evaluated in a custom simulation environment over 15,000 episodes. The Q-learning model achieved a high tracking accuracy of 92%, with a Root Mean Squared Error (RMSE) of 0.08.
The study also addressed "Hybrid System Instability." When the arm moves, it creates a dynamic moment that tilts the drone. The results demonstrated that while these movements cause immediate deviations in the UAV's angle of attack, the RL agent successfully compensates for these disturbances to keep the tool on its intended path.
Limitations and Future Directions
Despite the high accuracy, the authors noted that the learned trajectories can appear jittery due to the discretization of state and action spaces in standard Q-learning. Future work will focus on implementing more advanced techniques like Deep Q-Networks (DQN) or Model-Predictive Control (MPC) to achieve smoother transitions and handle more severe environmental disturbances, such as high winds.
Summary updated