DDPG-E2E: A Novel Policy Gradient Approach for End-to-End Communication Systems

Bolun Zhang, Nguyễn Văn Huynh, Dinh Thai Hoang, Diep N. Nguyen, Quoc‐Viet Pham

TL;DR

DDPG-E2E introduces a reinforcement learning framework that allows communication systems to be trained from end-to-end without needing a mathematical model of the wireless channel. By combining Deep Deterministic Policy Gradient (DDPG) with a convolutional neural network (CNN) architecture, the authors successfully scale deep-learning-based transceivers to handle large data blocks (up to 256 bits), significantly outperforming existing GAN-based and traditional coding methods.

What Problem Does This Paper Address?

Traditional communication systems are built from separate blocks (coding, modulation, estimation, etc.), each optimized individually. While effective, this "divide and conquer" approach isn't necessarily optimal for the system as a whole.

Recent "End-to-End" (E2E) approaches try to replace the entire chain with two neural networks: a transmitter and a receiver. However, they face two major hurdles:

  1. The Differentiability Gap: To train the transmitter using standard backpropagation, you need a differentiable mathematical model of the channel. In the real world, channels are "black boxes" that are often non-differentiable or unknown.
  2. The Curse of Dimensionality: Most E2E models use "one-hot encoding," which works for 4 or 8 bits but explodes exponentially as message length increases. Handling a standard 256-bit block is computationally impossible for these older architectures.

Methods

The authors propose a hybrid training strategy that treats the transmitter as a Reinforcement Learning (RL) agent and the receiver as a supervised learner.

  • DDPG for the Transmitter: The transmitter (the "Actor") learns to map bitstreams to complex-valued signals. Because it doesn't have a gradient from the channel, it uses the DDPG algorithm. It receives a "reward" from the receiver based on how accurately the data was recovered. This allows the system to optimize the transmitter even if the channel is a complete mystery.
  • Supervised Learning for the Receiver: The receiver is trained normally to minimize binary cross-entropy loss between the transmitted and estimated bits.
  • CNN Architecture: Instead of standard dense layers (MLPs), the authors use 1D-Convolutional Neural Networks. CNNs use weight sharing to recognize local patterns in bit sequences, which allows the model to scale to 256-bit blocks without the parameter explosion seen in previous E2E attempts.
  • Reward Feedback: The negative of the receiver's loss is sent back to the transmitter's "Critic" network, which estimates the "Q-value" (expected success) of the transmitter's actions.

Datasets

The study relies on extensive simulations using standard wireless channel models rather than static datasets:

  • Rayleigh Fading: Simulates environments with no direct line-of-sight (NLOS).
  • Rician Fading: Simulates environments with a strong direct line-of-sight (LOS) component.
  • 3GPP Spatial Channel Model: A complex, industry-standard model used to evaluate real-world cellular performance.
  • Block Lengths: Evaluated at 8, 128, and 256 bits.

Key Insights and Ideas

  • RL as a Bridge: The "aha moment" is using RL to bypass the need for a differentiable channel. By treating the channel and receiver as the "environment," the transmitter can learn through trial and error (guided by the Critic) rather than direct calculus.
  • Binary over One-Hot: By moving to binary encoding and CNNs, the authors break the 8-bit ceiling of previous E2E research. CNNs are naturally better at identifying the "local" relationships between bits in a stream.
  • Implicit Channel Learning: The system doesn't need to explicitly estimate the channel state (CSI). The feedback loop implicitly captures channel conditions, allowing the transceivers to adapt to Rayleigh or Rician fading automatically.

Key Results

  • Superior Accuracy: In Rayleigh fading scenarios with 256-bit blocks, DDPG-E2E achieved a significantly lower Block Error Rate (BLER) than GAN-based approaches and traditional LDPC (Low-Density Parity-Check) codes at high Signal-to-Noise Ratios (SNR).
  • Fast Convergence: The receiver loss converged to approximately $3 \times 10^{-5}$ within 2,000 iterations, outperforming baseline methods like Alternating Training (AT).
  • Robustness: The system performed nearly as well as "ideal" supervised models that have perfect knowledge of the channel, despite DDPG-E2E having zero prior channel information.
  • 3GPP Performance: Under the complex 3GPP model, the proposed solution showed the strongest resilience to noise compared to all other tested neural-based baselines.

Limitations and Future Work

  • Synchronization and Estimation: The paper notes that synchronization and explicit channel estimation were not the primary focus; the CNN handles these implicitly. In highly dynamic real-world hardware, explicit synchronization might still be necessary.
  • Computational Overhead: While CNNs are more efficient than MLPs for large blocks, training an Actor-Critic model (DDPG) is generally more computationally intensive than simple supervised learning.
  • Future Directions: The authors suggest extending this to multi-user scenarios (MIMO) and testing the architecture on actual Software Defined Radio (SDR) hardware to validate performance outside of simulations.

Summary updated

Abstract

The End-to-end (E2E) learning-based approach has great potential to reshape the existing communication systems by replacing the transceivers with deep neural networks. To this end, the E2E learning approach needs to assume the availability of prior channel information to mathematically formulate a differentiable channel layer for the backpropagation (BP) of the error gradients, thereby jointly optimizing the transmitter and the receiver. However, accurate and instantaneous channel state information is hardly obtained in practical wireless communication scenarios. Moreover, the existing E2E learning-based solutions exhibit limited performance in data transmissions with large block lengths. In this article, these practical issues are addressed by our proposed deep deterministic policy gradient-based E2E communication system. In particular, the proposed solution utilizes a reward feedback mechanism to train both the transmitter and the receiver, which alleviates the information loss of error gradients during BP. In addition, a convolutional neural network (CNN)-based architecture is developed to mitigate the curse of dimensionality problem when transmitting messages with large block lengths. Extensive simulations then demonstrate that our proposed solution can not only jointly train the transmitter and the receiver simultaneously without requiring the prior channel knowledge but also can obtain significant performance improvement on block error rate compared to state-of-the-art solutions.
BibTeX
@article{zhang2025ddpg,
  author = {Bolun Zhang and Nguyễn Văn Huynh and Dinh Thai Hoang and Diep N. Nguyen and Quoc‐Viet Pham},
  title = {{DDPG-E2E: A Novel Policy Gradient Approach for End-to-End Communication Systems}},
  journal = {IEEE Trans. Cogn. Commun. Netw},
  year = {2025},
  doi = {10.1109/tccn.2024.3485648},
}
Figure