Security State Estimation for Cyber-Physical Systems against DoS Attacks via Reinforcement Learning and Game Theory

Zengwang Jin, Shuting Zhang, Yanyan Hu, Yanning Zhang, Changyin SunActuators 2022 · DOI

TL;DR

This paper tackles the critical challenge of securing Cyber-Physical Systems (CPSs) against Denial-of-Service (DoS) attacks by framing the interaction between sensors and attackers as a game. They propose a novel reinforcement learning approach that learns optimal, Nash equilibrium strategies for both parties, considering different channel reliability scenarios.

What Problem Does This Paper Address?

Cyber-Physical Systems (CPSs) are increasingly integrated into critical infrastructure, relying heavily on wireless sensors for data collection. However, these systems are vulnerable to cyberattacks, particularly DoS attacks, which can disrupt communication channels and compromise the accuracy of state estimation. Existing methods often focus on either attack detection or system stability, but there's a need for a framework that models the dynamic, strategic interplay between attackers and defenders to optimize security policies. This paper addresses this by considering the optimal policy selection for both the sensor (defender) and the attacker, aiming to achieve a secure state estimation in the face of these adversarial actions.

Methods

The core of this paper lies in combining game theory and reinforcement learning.

  • Two-Player Zero-Sum Game: The interaction between the sensor and the attacker is modeled as a two-player zero-sum game. This means one player's gain is the other's loss, reflecting their opposing objectives (sensor wants accurate estimation, attacker wants to disrupt it). The goal is to find Nash equilibrium strategies, where neither player can improve their outcome by unilaterally changing their strategy.
  • Kalman Filter for State Estimation: A standard Kalman filter is used for recursive state estimation. The paper considers how DoS attacks impact this estimation process, particularly the error covariance.
  • Reinforcement Learning (Q-Learning): A Q-learning algorithm is employed to enable the sensor and attacker to learn optimal strategies through interaction.
  • ε-greedy policy: This standard RL technique is used to balance exploration (trying new actions) and exploitation (using known good actions). The paper mentions an "improved" ε-greedy policy, suggesting a dynamic adjustment of ε.
  • Two Scenarios: The RL algorithm is designed and evaluated for two distinct scenarios:
  • Reliable Channel: Packet loss occurs only due to DoS attacks.
  • Unreliable Channel: Packet loss can occur due to DoS attacks or other factors like signal degradation, fading, or congestion. This introduces more complex packet loss probabilities.
  • Linear Programming: Used to calculate the Nash equilibrium strategies from the learned Q-values.

Datasets

Not discussed in the paper. The paper uses simulated CPS parameters for its experiments.

Key Insights and Ideas

  • Game Theory as a Security Framework: Framing CPS security as a game between an attacker and a defender provides a powerful lens for understanding and optimizing adversarial interactions.
  • Dynamic Policy Adjustment: Reinforcement learning allows for dynamic adaptation of strategies, which is crucial in evolving cyberattack landscapes, unlike static rule-based systems.
  • Channel Reliability Matters: The distinction between reliable and unreliable channels significantly impacts optimal strategies and overall system security. Unreliable channels necessitate more cautious (and potentially costly) defensive actions.
  • Nash Equilibrium as Optimal Outcome: Achieving Nash equilibrium ensures a stable state where both the sensor and attacker are playing their best response, leading to a predictable and manageable security posture.

Key Results

The simulation results demonstrate that the proposed reinforcement learning algorithm:

  • Quickly Converges: The Q-value matrix converges to stable values in a finite number of steps for both reliable and unreliable channels.
  • Achieves Nash Equilibrium: The algorithm successfully identifies Nash equilibrium policies for both the sensor and the attacker.
  • Reduces Packet Loss: In the reliable channel scenario, the probability of two consecutive packet losses was 0.04%, and single packet loss was 2.44%. For the unreliable channel, these figures were 0.42% and 3.76%, respectively. While packet loss is higher in the unreliable case, the algorithm still minimizes it compared to random actions.
  • Efficient Computation: The running time for the algorithm was found to be 0.277 s for the reliable channel and 0.489 s for the unreliable channel, indicating reasonable computational efficiency.

Limitations and Future Work

  • Single-Sensor, Single-Attacker Focus: The current model focuses on a single sensor and a single attacker. Future work aims to extend this to multi-sensor and multi-attacker scenarios.
  • DDoS Attacks: The paper acknowledges that extending the model to Distributed Denial-of-Service (DDoS) attacks, which involve multiple attack sources, is a significant future direction. This would introduce complexities in coordinating multiple attackers and defending against them.
  • Model Expansion: Investigating the model's applicability to different types of attacks beyond DoS is also a future consideration.
  • Resource Constraints: While the paper mentions resource allocation, further exploration of limited bandwidth and energy constraints for optimizing strategies is suggested.

Summary updated

Abstract

This paper addressed the optimal policy selection problem of attacker and sensor in cyber-physical systems (CPSs) under denial of service (DoS) attacks. Since the sensor and the attacker have opposite goals, a two-player zero-sum game is introduced to describe the game between the sensor and the attacker, and the Nash equilibrium strategies are studied to obtain the optimal actions. In order to effectively evaluate and quantify the gains, a reinforcement learning algorithm is proposed to dynamically adjust the corresponding strategies. Furthermore, security state estimation is introduced to evaluate the impact of offensive and defensive strategies on CPSs. In the algorithm, the ε-greedy policy is improved to make optimal choices based on sufficient learning, achieving a balance of exploration and exploitation. It is worth noting that the channel reliability factor is considered in order to study CPSs with multiple reasons for packet loss. The reinforcement learning algorithm is designed in two scenarios: reliable channel (that is, the reason for packet loss is only DoS attacks) and unreliable channel (the reason for packet loss is not entirely from DoS attacks). The simulation results of the two scenarios show that the proposed reinforcement learning algorithm can quickly converge to the Nash equilibrium policies of both sides, proving the availability and effectiveness of the algorithm.
BibTeX
@inproceedings{jin2022security,
  author = {Zengwang Jin and Shuting Zhang and Yanyan Hu and Yanning Zhang and Changyin Sun},
  title = {{Security State Estimation for Cyber-Physical Systems against DoS Attacks via Reinforcement Learning and Game Theory}},
  booktitle = {Actuators},
  year = {2022},
  doi = {10.3390/act11070192},
}
Figure