Balancing Performance and Cost for Two-Hop Cooperative Communications: Stackelberg Game and Distributed Multi-Agent Reinforcement Learning

Yuanzhe Geng, Erwu Liu, Wei Ni, Rui Wang, Yan Liu, Hao Xu, Chen Cai, Abbas Jamalipour

Abstract

In decentralized wireless networks, optimizing the trade-off between signal quality and economic cost is a complex challenge, particularly when nodes have conflicting interests and operate with outdated channel information. This paper proposes a framework combining a Stackelberg game model with Multi-Agent Reinforcement Learning (MARL) to achieve a near-optimal equilibrium in two-hop relay networks. By modeling the relay nodes as a collective leader and the source as a follower, the authors demonstrate that a Deep Deterministic Policy Gradient (DDPG) approach can converge to within 2.9% of the theoretical optimum, even under realistic conditions of delayed Channel State Information (CSI).

Introduction

Two-hop cooperative communication relies on intermediate relay nodes to extend the reach and reliability of a source signal to a destination. Traditional optimization methods often assume a central controller with perfect knowledge of the network state. However, modern decentralized networks consist of independent agents with distinct objectives: source nodes seek to maximize capacity at minimum cost, while relay nodes aim to maximize revenue for their power expenditure.

This conflict of interest, compounded by the rapid fluctuations of wireless channels—which often render Channel State Information (CSI) outdated by the time decisions are implemented—necessitates a robust, distributed solution. This paper addresses these challenges by framing the interaction as a Stackelberg game and solving it using a MARL-based approach.

The Stackelberg Game Framework

The interaction between the source and the relays is modeled as a leader-follower game. In this hierarchy, the Relay Alliance acts as the leader, setting a price per unit of power. The Source, as the follower, determines its power purchase based on this price to maximize its utility (capacity minus cost).

The authors provide mathematical proofs establishing that a unique Nash Equilibrium exists for this game under perfect information conditions. A critical insight from this modeling is the "Power of Alliance": by coordinating as a single entity rather than competing, relays prevent the source from driving prices down, thereby securing higher revenue while maintaining network performance.

Distributed Multi-Agent Reinforcement Learning

To bridge the gap between theoretical models and real-world uncertainty, the paper employs a Multi-Agent Deep Deterministic Policy Gradient (DDPG) algorithm. This approach is designed to handle continuous action spaces and unknown utility functions.

Key Components of the MARL Approach:

  1. Simultaneous Training: Two agents—representing the Relay Alliance and the Source—train concurrently to find the equilibrium.
  2. Prioritized Experience Replay: This technique allows the agents to learn more effectively from significant past interactions, improving sample efficiency.
  3. Robustness to Outdated CSI: Crucially, the agents make decisions based on CSI from the previous time slot. This mimics the inherent delays in real-world feedback loops. The MARL agents learn to identify underlying channel patterns rather than simply reacting to instantaneous, and potentially stale, snapshots.

Experimental Evaluation

The framework was evaluated using numerical simulations based on a Gaussian Markov block fading autoregressive model to simulate time-varying wireless environments. The simulation parameters included a source, a destination, and $K$ relays, with a maximum power constraint ($P_{max} = 2.0W$) and a price cap ($C_{max} = 10.0$).

Results and Performance

The proposed MARL method demonstrated significant advantages over traditional and single-agent learning baselines:

  • Proximity to Optimum: The system achieved performance within 2.9% of the theoretical "Game-Based Solution" (which assumes perfect, instantaneous information).
  • Baseline Comparison: The DDPG-based MARL outperformed single-agent learning (LGMS) and discretized multi-agent approaches (DQN-MARL). For instance, at a source power of 1.0W, the proposed method reached a channel capacity of approximately 1.08 bits/s/Hz, whereas competing learning methods reached only 0.8–0.9 bits/s/Hz.
  • Convergence Efficiency: The agents reached a stable equilibrium in approximately 75 training episodes, indicating high sample efficiency for a multi-agent environment.

Conclusion and Future Work

This research demonstrates that MARL is a powerful tool for solving complex game-theoretic problems in engineering systems where agents have hidden goals and information is imperfect. The proposed framework effectively balances the trade-off between performance and cost in two-hop networks, showing remarkable resilience to CSI delays.

Future research may explore the scalability of the "Relay Alliance" in highly mobile or massive-scale networks, where coordination overhead might increase. Additionally, extending the model to multi-hop topologies or environments with significant inter-source interference remains a promising direction for further study.

The implementation of this framework is available on GitHub.

Summary updated

BibTeX
@article{geng2024balancing,
  author = {Yuanzhe Geng and Erwu Liu and Wei Ni and Rui Wang and Yan Liu and Hao Xu and Chen Cai and Abbas Jamalipour},
  title = {{Balancing Performance and Cost for Two-Hop Cooperative Communications: Stackelberg Game and Distributed Multi-Agent Reinforcement Learning}},
  journal = {IEEE Trans. Cogn. Commun. Netw.},
  year = {2024},
  doi = {10.1109/tccn.2024.3400516},
}
Figure