Extremum and Nash Equilibrium Seeking with Delays and PDEs: Designs & Applications

Tiago Roux Oliveira, Miroslav Krstic, Tamer BasararXiv 2024 · DOI · arXiv

Abstract

This paper extends model-free Extremum Seeking (ES) and Nash Equilibrium Seeking (NES) to systems governed by infinite-dimensional dynamics, including time delays and Partial Differential Equations (PDEs). By combining perturbation-based optimization with boundary control and predictor feedback, the authors demonstrate that agents can find optimal operating points or game-theoretic equilibria even when their actuators involve complex transport, diffusion, or wave dynamics. Rigorous stability proofs are provided using averaging theory for infinite dimensions and small-gain analysis.

Introduction

In classical optimization, an agent's action is typically assumed to have an immediate effect on their payoff. However, real-world engineering often involves "infinite-dimensional" actuators. For instance, in oil drilling, surface commands propagate through a mile-long drill string (wave PDE), and in chemical reactors, inputs diffuse through a medium (parabolic PDE). These systems present two major challenges: the payoff functions are often unknown (model-free requirement), and the inherent "lag" or "smearing" of the actuator can cause instability if not properly compensated.

Methodology: Predictor Feedback and Boundary Control

The proposed framework builds upon standard ES, which uses sinusoidal "dither" signals to estimate gradients. To handle infinite-dimensional dynamics, the authors integrate advanced control techniques:

  1. Predictor Feedback: For systems with pure time delays (transport PDEs), predictor-based update laws "look ahead" to compensate for dead time.
  2. PDE Boundary Control: For systems governed by heat or wave equations, boundary controllers ensure that dither signals reach the payoff map in a usable form, effectively "inverting" the actuator dynamics.
Control block diagram for a single agent using predictor feedback to compensate for input delays.

The stability of these designs is verified through mathematical averaging techniques specifically adapted for Hilbert spaces.

Nash Equilibrium Seeking in Complex Environments

The paper generalizes single-agent ES to multi-player Nash Equilibrium Seeking (NES). A key insight is that NES can remain stable even when players are governed by heterogeneous physics—for example, one player facing a transport delay while another deals with heat diffusion.

Architecture for a two-player noncooperative game where each player's actions are governed by heat PDE dynamics.

Using small-gain theorems, the authors prove that the coupled system converges to a neighborhood of the Nash Equilibrium as long as the interference between players (off-diagonal terms in the Hessian) is sufficiently small relative to their own payoff sensitivity.

Applications and Key Results

The framework's versatility is demonstrated across several domains:

  • Traffic Control: Maximizing flow through bottlenecks using the Lighthill-Whitham-Richards model.
  • Biomedical Engineering: Optimizing neuromuscular electrical stimulation (NMES) for rehabilitation despite inherent physiological delays.
  • Manufacturing: Controlling the melt-front in additive manufacturing processes (the Stefan problem).

Numerical simulations of a heterogeneous duopoly confirm that convergence is robust to the specific type of actuation physics involved. The paper provides explicit bounds on the residual error, showing how dither frequency and amplitude influence final accuracy.

Limitations and Future Work

While the payoff is model-free, the current design assumes the parameters of the PDE actuator (e.g., delay length or diffusion coefficient) are known. Future research may explore "PDE-adaptive" seeking to remove this requirement. Additionally, the small-gain assumption for stability is sufficient but potentially conservative; real-world systems might tolerate stronger coupling. Extending these deterministic results to stochastic extremum seeking also remains an open area for investigation.

Summary updated

BibTeX
@misc{oliveira2024extremum,
  author = {Tiago Roux Oliveira and Miroslav Krstic and Tamer Basar},
  title = {{Extremum and Nash Equilibrium Seeking with Delays and PDEs: Designs \& Applications}},
  year = {2024},
  eprint = {2411.13234},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.13234},
}
Figure