A Dynamics and Task Decoupled Reinforcement Learning Architecture for High-Efficiency Dynamic Target Intercept

Dora D. Liu, Liang Hu, Qi Zhang, Tangwei Ye, Usman Naseem, Zhong Yuan LaiAAAI 2023 · OpenReview

Abstract

Due to the flexibility and ease of control, unmanned aerial vehicles (UAVs) have been increasingly used in various scenarios and applications in recent years. Training UAVs with reinforcement learning (RL) for a specific task is often expensive in terms of time and computation. However, it is known that the main effort of the learning process is made to fit the low-level physical dynamics systems instead of the high-level task itself. In this paper, we study to apply UAVs in the dynamic target intercept (DTI) task, where the dynamics systems equipped by different UAV models are correspondingly distinct. To this end, we propose a dynamics and task decoupled RL architecture to address the inefficient learning procedure, where the RL module focuses on modeling the DTI task without involving physical dynamics, and the design of states, actions, and rewards are completely task-oriented while the dynamics control module can adaptively convert actions from the RL module to dynamics signals to control different UAVs without retraining the RL module. We show the efficiency and efficacy of our results in comparison and ablation experiments against state-of-the-art methods.

Introduction: The Challenge of Dynamic Target Intercept

Intercepting a moving target in 3D space—Dynamic Target Intercept (DTI)—is a critical yet complex task for Unmanned Aerial Vehicles (UAVs). Traditional reactive control systems often suffer from a lag effect: by the time an interceptor reaches the target's observed position, the target has already moved. This leads to a perpetual chase, often referred to as a version of Zeno’s paradox or the "Achilles and the tortoise" problem.

Comparison of non-predictive (reactive) and predictive path planning strategies.

While Reinforcement Learning (RL) can model complex behaviors, standard end-to-end RL approaches face two major hurdles in DTI:

  1. Complexity Overload: Simultaneously learning high-level strategy and low-level flight physics (thrust, torque, RPM) is computationally expensive and slow.
  2. Poor Generalization: RL agents trained on specific drone dynamics (e.g., a quadcopter) typically require complete retraining to operate on different hardware (e.g., a fixed-wing drone).

The DTTD Architecture

To overcome these limitations, we propose the DTTD architecture (Decoupling Trajectory prediction, Task learning, and Dynamics control). This framework separates the high-level "where to go" strategy from the low-level "how to fly" physics.

The DTTD architecture, illustrating the decoupling of trajectory prediction, task learning, and dynamics control across offline and online phases.

The architecture consists of three primary modules:

  1. Target’s Trajectory Prediction Module: Utilizing Multivariate Time Series (MTS) models such as Seq2seq or LSTM, this module predicts future target waypoints rather than reacting to current positions.
  2. Task’s Interception Path Planning Module: This "brain" uses Soft Actor-Critic (SAC) to determine the optimal path to a predicted future waypoint. By operating solely on coordinates and velocities, it remains dynamics-independent.
  3. Agent’s Dynamics Control Module: This module translates high-level movement commands into hardware-specific control signals (e.g., motor thrust) via an A2C Adaptor and ODE solvers.

To enhance training, we introduced an Energy-based Reward Function (ERF), providing progressive feedback based on proximity, efficiency, and time, rather than a binary success/failure signal.

Experimental Evaluation

The system was evaluated in simulated environments against targets following complex, non-linear trajectories, including Spiral and Lemniscate (figure-eight) paths.

Key Findings

  • Decoupling Efficiency: Stripping physics from the RL loop allows the agent to focus on the geometry of interception, significantly speeding up training.
  • Predictive Superiority: Successful interception relies on identifying the earliest reachable future waypoint. The Seq2seq model outperformed standard LSTMs by providing a multi-step look-ahead for smoother planning.
  • Universal Adaptability: The dynamics-independent RL module can be deployed across different UAV models by simply swapping the Dynamics Control Module.
  • Performance: DTTD achieved a 1.00 Weighted Rank Sum Ratio (WRSR) in both spiral and lemniscate tests, consistently outperforming baselines like Extended Kalman Filters (EKF) and Proportional Guidance (ZEM).

Conclusion and Future Work

The DTTD framework demonstrates that decoupling task logic from physical dynamics significantly improves learning efficiency and cross-platform adaptability in DTI tasks. Future research will address challenges in partially observable environments, explore model-free dynamics controllers for drones with unknown physical properties, and integrate obstacle avoidance into the decoupled architecture.

Summary updated

Abstract

Due to the flexibility and ease of control, unmanned aerial vehicles (UAVs) have been increasingly used in various scenarios and applications in recent years. Training UAVs with reinforcement learning (RL) for a specific task is often expensive in terms of time and computation. However, it is known that the main effort of the learning process is made to fit the low-level physical dynamics systems instead of the high-level task itself. In this paper, we study to apply UAVs in the dynamic target intercept (DTI) task, where the dynamics systems equipped by different UAV models are correspondingly distinct. To this end, we propose a dynamics and task decoupled RL architecture to address the inefficient learning procedure, where the RL module focuses on modeling the DTI task without involving physical dynamics, and the design of states, actions, and rewards are completely task-oriented while the dynamics control module can adaptively convert actions from the RL module to dynamics signals to control different UAVs without retraining the RL module. We show the efficiency and efficacy of our results in comparison and ablation experiments against state-of-the-art methods.
BibTeX
@inproceedings{liu2023dynamics,
  author = {Dora D. Liu and Liang Hu and Qi Zhang and Tangwei Ye and Usman Naseem and Zhong Yuan Lai},
  title = {{A Dynamics and Task Decoupled Reinforcement Learning Architecture for High-Efficiency Dynamic Target Intercept}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=pqT5CqguH9},
}
Figure