Robust, Deep, and Reinforcement Learning for Management of Communication and Power Networks

Alireza Sadeghi

Author: Alireza Sadeghi

Abstract

This research develops data-driven machine learning (ML) algorithms designed to manage and optimize next-generation cyber-physical systems (CPS). These systems—ranging from 5G communication networks to smart power grids—require control and monitoring schemes that guarantee robustness, scalability, and situational awareness. The work first establishes principled methods to make generic ML models robust against distributional uncertainties and adversarial data, specifically focusing on parametric models. Using a distributionally robust learning framework, the research minimizes worst-case expected loss over an ambiguity set quantified via Wasserstein distance. These frameworks are then extended to semi-supervised learning and applied to network entities using deep reinforcement learning (DRL). Finally, the thesis addresses sustainable distribution grids, delegating reactive power compensation to utility-owned devices and smart inverters to manage the volatility of renewable energy sources.

Introduction

Modern infrastructure relies increasingly on Cyber-Physical Systems (CPS) to manage the flow of data and electricity. However, traditional machine learning models often falter when deployed in these real-world environments. Most ML models assume that training data and testing data are drawn from the same distribution (the i.i.d. assumption). In critical infrastructure, this assumption is frequently violated by sensor drift, environmental shifts, or malicious adversarial attacks.

This paper explores the development of robust, scalable, and physics-aware learning models capable of maintaining performance under uncertainty. By bridging the gap between rigorous mathematical optimization and deep learning, this work provides a roadmap for deploying AI in high-stakes engineering environments.

Distributionally Robust Learning

A core contribution of this work is the development of Distributionally Robust Optimization (DRO) frameworks. Rather than optimizing for a single dataset, the proposed approach considers a "ball" of possible distributions—an ambiguity set—defined by the Wasserstein distance. The goal is to minimize the loss against the worst-case scenario within this set.

To make this computationally feasible, the author leverages mathematical duality. This transformation converts an infinite-dimensional optimization problem into a tractable dual problem. To solve these minimax problems efficiently, the research introduces Stochastic Proximal Gradient Descent-Ascent (SPGDA). This algorithm allows for lightweight, single-step updates, making robust training practical for large-scale models.

Intelligent Communication Networks

In the realm of 5G and next-generation wireless networks, the research focuses on the challenge of content caching. Managing what data to store at the "edge" of a network is a complex sequential decision-making problem. The author treats this as a two-timescale problem:

  1. Slow Timescale: Using Deep Reinforcement Learning (DRL), specifically Deep Q-Networks (DQN), to make storage decisions based on evolving content popularity.
  2. Fast Timescale: Using optimization techniques to handle immediate data fetching requests.

This hybrid approach ensures that the network can adapt to non-stationary demand without the overhead of constant re-optimization.

Physics-Aware Power Systems

Power grids present a unique challenge: they are governed by strict physical laws (Kirchhoff’s laws) that purely data-driven models often ignore. To address this, the research introduces Gauss-Newton Unrolled Neural Networks (GNU-GNN). By "unrolling" the classical Gauss-Newton optimization algorithm into a Graph Neural Network architecture, the model inherits the inductive bias of the grid's topology.

This architecture is applied to:

  • State Estimation: Monitoring the real-time status of the grid.
  • Voltage Regulation: Managing reactive power compensation through smart inverters to mitigate the fluctuations caused by solar and wind energy.

Experimental Validation

The proposed methods were validated across several domains:

  • Adversarial Robustness: On MNIST and Fashion-MNIST, the DRO framework maintained high accuracy against Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks that caused standard models to fail.
  • Grid Efficiency: Using IEEE 118-bus and 123-bus benchmarks, the GNU-GNN achieved state estimation speeds three orders of magnitude faster than traditional iterative solvers, with superior accuracy.
  • Renewable Integration: Using data from the Smart* Project and GEFC 2012, the DRL-based controllers successfully stabilized voltage levels in feeders with high renewable penetration, outperforming standard heuristic controllers.

Conclusion and Future Work

This dissertation demonstrates that robustness and efficiency are not mutually exclusive in CPS management. By combining distributionally robust optimization with deep reinforcement learning and physics-based constraints, we can create AI systems that are both fast and reliable.

Future research directions include scaling these methods to even larger networks through Multi-agent Reinforcement Learning (MARL) and incorporating formal "fairness" constraints to ensure that robust models do not inadvertently disadvantage specific nodes or subpopulations within a network.

Summary updated

Abstract

This thesis develops data-driven machine learning algorithms to managing and optimizing the next-generation highly complex cyberphysical systems, which desperately need ground-breaking control, monitoring, and decision making schemes that can guarantee robustness, scalability, and situational awareness. The present thesis first develops principled methods to make generic machine learning models robust against distributional uncertainties and adversarial data. Particular focus will be on parametric models where some training data are being used to learn a parametric model. The developed framework is of high interest especially when training and testing data are drawn from "slightly" different distribution. We then introduce distributionally robust learning frameworks to minimize the worst-case expected loss over a prescribed ambiguity set of training distributions quantified via Wasserstein distance. Later, we build on this robust framework to design robust semi-supervised learning over graph methods. The second part of this thesis aspires to fully unleash the potential of next-generation wired and wireless networks, where we design "smart" network entities using (deep) reinforcement learning approaches. Finally, this thesis enhances the power system operation and control. Our contribution is on sustainable distribution grids with high penetration of renewable sources and demand response programs. To account for unanticipated and rapidly changing renewable generation and load consumption scenarios, we specifically delegate reactive power compensation to both utility-owned control devices (e.g., capacitor banks), as well as smart inverters of distributed generation units with cyber-capabilities.
BibTeX
@misc{sadeghi2022robust,
  author = {Alireza Sadeghi},
  title = {{Robust, Deep, and Reinforcement Learning for Management of Communication and Power Networks}},
  year = {2022},
  eprint = {2202.05395},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.05395},
}
Figure