Reinforcement Learning for Interference Avoidance Game in RF-Powered Backscatter Communications

Ali Rahmati, Huaiyu DaiICC 2019 · DOI

TL;DR

This paper tackles interference in battery-free RF-powered backscatter communication systems by modeling the interaction between a user and a smart interferer as a dynamic game. They propose a Q-learning approach, enhanced with "hotbooting," to enable the user to learn optimal transmission strategies in real-time, significantly improving performance and convergence speed compared to traditional methods.

What Problem Does This Paper Address?

RF-powered backscatter communication is a promising technology for low-power, battery-free devices like those in the Internet of Things (IoT) and wireless sensor networks (WSNs). These devices harvest energy from ambient RF signals and use it to transmit data by reflecting these signals. However, this communication method is inherently vulnerable to interference and jamming, which can degrade the quality of the backscattered signal and hinder communication. A key challenge arises when a "smart interferer" actively tries to disrupt the user's communication. In practical scenarios, the user often lacks information about the interferer's strategy or the overall system conditions (like signal-to-interference-plus-noise ratio, SINR). This necessitates a dynamic approach where the user can learn to adapt its transmission strategy through trial and error.

Methods

The paper formulates the interaction between the user and a smart interferer as a Stackelberg game. In this game, the user acts as the "leader," deciding its backscattering time, while the smart interferer is the "follower," choosing its jamming power to minimize the user's utility.

The core of their solution lies in Q-learning, a reinforcement learning algorithm. Q-learning allows an agent (the user) to learn an optimal policy by taking actions in an environment and receiving rewards (or penalties). Here, the user learns to choose its backscattering time to maximize its utility (number of transmitted bits minus costs).

To address the slow convergence of standard Q-learning, they introduce "hotbooting" Q-learning. This method initializes the Q-tables (which store the learned values of state-action pairs) using pre-collected data from similar environments. This "warm start" significantly speeds up the learning process by reducing the initial random exploration phase.

Datasets

Not discussed in the paper. The evaluation is based on simulations with defined parameters.

Key Insights and Ideas

  • Game Theory for Interference: Framing the user-interferer interaction as a Stackelberg game provides a robust theoretical foundation for understanding and optimizing their dynamic relationship.
  • Utility Function Design: Defining the utility not just by SINR but by the actual number of transmitted bits (considering backscattering time and costs) offers a more realistic model for backscatter communication.
  • Learning in the Dark: The Q-learning approach is crucial because it allows the user to learn optimal strategies even without prior knowledge of the interferer's capabilities or the exact channel conditions.
  • Hotbooting for Speed: The "hotbooting" technique is a clever way to accelerate reinforcement learning by leveraging prior experience, making the system more practical for real-world deployment where rapid adaptation is key.

Key Results

  • The proposed Q-learning based dynamic solution significantly improves the user's utility compared to random or fixed backscattering time strategies.
  • Hotbooting Q-learning demonstrates a 31% improvement in convergence speed compared to standard Q-learning.
  • The user's utility decreases as the distance to the HAP increases, which is an expected but important validation of the model.
  • Increasing the cost associated with backscattering time directly reduces the user's utility, highlighting the importance of balancing transmission time with operational costs.

Limitations and Future Work

  • Non-linearity: The paper notes that obtaining a closed-form solution for the optimal backscattering time is challenging due to non-linear equations, requiring numerical methods.
  • Quantization: The user quantizes the received SINR into discrete levels, which is a simplification.
  • Future Work: The authors suggest exploring the impact of user and interferer mobility and location on the game's equilibrium. They also propose using deep Q-learning to handle a larger number of variables and reduce complexity, especially when dealing with more complex scenarios or a larger state space.

Summary updated

BibTeX
@inproceedings{rahmati2019reinforcement,
  author = {Ali Rahmati and Huaiyu Dai},
  title = {{Reinforcement Learning for Interference Avoidance Game in RF-Powered Backscatter Communications}},
  booktitle = {ICC},
  year = {2019},
  doi = {10.1109/icc.2019.8761145},
}
Figure