Communication Emitter Motion Behavior's Cognition Based on Deep Reinforcement Learning

Yufan Ji, Jiang Wang, Weilu Wu, Lunwen Wang, Chuang Peng, Hao ShaoIEEE Access 2021 · DOI

Introduction: Moving Beyond Simple Tracking

In signal processing and electronic warfare, tracking the coordinates of an object is a well-established capability. However, knowing where an object is doesn't necessarily explain why it is moving that way. "Cognizing" behavior involves understanding the intent or environmental factors—such as signal interference or physical obstacles—that cause a change in movement.

Traditional methods often treat behavior identification as a discrete classification problem or rely on subjective manual feature engineering. These approaches struggle with the multi-dimensional, continuous nature of emitter data, which includes position, velocity, and communication frequency.

Conceptual diagram of emitter route deviation due to external interference.

Proposed Methodology: DDPG and Geographic Attention

To address these challenges, this research proposes a framework based on the Deep Deterministic Policy Gradient (DDPG) algorithm. DDPG is an actor-critic model specifically designed for continuous action spaces, making it ideal for physical movement and signal tuning.

The authors developed two primary models:

  1. DDPG-BC (Behavior Cognition): Maps physical parameters like velocity, direction, and frequency to specific cognition results (e.g., "Normal" vs. "Disturbed").
  2. DDPG+A-BC (with Attention): This model introduces a "hard" attention mechanism. By incorporating prior geographic knowledge—such as known interference zones—the model focuses its exploration on these critical regions, significantly reducing the search space for the neural network.
The overall cognition process flow, integrating raw data processing with the DDPG-based attention mechanism.

Datasets and Experimental Setup

Due to the lack of public datasets for emitter motion behavior, the researchers created two simulation environments using real-world trajectories:

  • Flightradar24 Data: 140 groups of commercial flight trajectories.
  • Geolife Project Data: Over 6,800 sampling points of complex pedestrian trajectories from Microsoft Research Asia. In both cases, communication frequency data was manually augmented to simulate realistic emitter behavior under different conditions.

Key Results and Performance

The inclusion of the attention mechanism proved transformative:

  • Accuracy: On the flight dataset, DDPG+A-BC achieved 99.03% accuracy, compared to 90.43% for the standard DDPG model. On the more complex Geolife dataset, it maintained a high accuracy of 92.5%.
  • Efficiency: The attention mechanism drastically accelerated training. The DDPG+A-BC model reached stable cognition in just 19 episodes, whereas the baseline required 235 episodes.
  • Benchmarking: The proposed approach significantly outperformed discrete algorithms like Double-DQN and experience-based frameworks like EOIT, showing an average accuracy lead of over 30% against the latter.

Conclusion and Future Directions

This research demonstrates that Deep Reinforcement Learning, when combined with geographic attention, provides a robust and objective way to understand emitter behavior in continuous spaces. While the results are promising, future work is needed to test these models in noisier, real-world RF environments and to develop "soft" attention mechanisms that are less dependent on rigid prior geographic information.

Summary updated

BibTeX
@article{ji2021communication,
  author = {Yufan Ji and Jiang Wang and Weilu Wu and Lunwen Wang and Chuang Peng and Hao Shao},
  title = {{Communication Emitter Motion Behavior's Cognition Based on Deep Reinforcement Learning}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2020.3047845},
}
Figure