Rateless Autoencoder Codes: Trading off Decoding Delay and Reliability

Vukan Ninkovic, Dejan Vukobratovic, Christian Häger, Henk Wymeersch, Alexandre Graell i Amat

TL;DR

This paper introduces Rateless Autoencoder (AE) Codes, a deep-learning-based approach to error correction designed for "dying channels" where transmissions might be cut short. By incorporating specialized dropout strategies during training, the authors create codes that allow for a graceful trade-off between decoding delay and reliability, significantly outperforming conventional autoencoders when only a partial codeword is received.

What Problem Does This Paper Address?

In ideal communication theory, we assume a receiver gets the entire codeword before decoding. However, real-world "dying channels" are unpredictable. A transmission might be interrupted due to a satellite moving out of range, a sensor running out of harvested energy, or a primary user reclaiming a frequency in cognitive radio.

Standard fixed-rate codes (including standard neural autoencoders) are "all-or-nothing": they are optimized for a specific block length. If the channel "dies" and the receiver only gets 70% of the symbols, the error rate usually skyrockets. While classical "rateless" codes (like Spinal or Fountain codes) exist to solve this, they can be difficult to optimize for very short block lengths or specific hardware constraints. This paper seeks to bring the flexibility of rateless coding to the end-to-end optimized world of Deep Learning.

Methods

The authors build upon the standard Deep Autoencoder (AE) for the physical layer, where an encoder neural network maps a message to a sequence of symbols, and a decoder network maps the noisy received symbols back to the message.

The core innovation is the introduction of Channel Dropout Strategies during the training phase:

  1. Model 1: Tail Erasures (The "Dying" Channel): During training, the authors apply a dropout mask that zeros out the end of a codeword. By varying how many symbols are "dropped" in each training batch, the encoder is forced to learn a representation where the message can be recovered (with varying levels of certainty) from just the first few symbols.
  2. Model 2: Random Erasures: Here, symbols are dropped randomly across the codeword, simulating a channel with intermittent deep fades or packet loss.
  3. Training Logic: The system is trained end-to-end using the Adam optimizer and a cross-entropy loss function. Crucially, the dropout is randomized batch-by-batch, so the network never knows exactly how many symbols it will have available, forcing it to prioritize information density.

Datasets

The paper uses synthetically generated data, which is standard for physical layer communications research:

  • Message Space: $M = 4096$ unique messages (equivalent to $k=12$ bits).
  • Codeword Length: $n = 24$ symbols.
  • Training/Test Sets: $10^5$ messages for training and $10^6$ for testing to ensure statistical significance in Block Error Rate (BLER) measurements.
  • Channel: Additive White Gaussian Noise (AWGN) combined with the erasure models mentioned above.

Key Insights and Ideas

  • Dropout as a Structural Tool: Usually, dropout is used in deep learning to prevent overfitting. Here, it is used as a physical layer proxy. By "dropping" codeword symbols during training, the authors bake the requirement for rate-compatibility directly into the weights of the neural network.
  • Information Pre-loading: In the "Tail Erasure" model, the encoder naturally learns to put the most critical information in the earlier symbols of the codeword.
  • Graceful Degradation: Unlike conventional AEs, which show a "cliff effect" in performance if symbols are missing, Rateless AEs show a smooth, predictable improvement in reliability for every additional symbol received.
  • Robustness to Mismatch: The authors found that even if the "dropout distribution" used during training doesn't perfectly match the real-world channel statistics, the Rateless AE still significantly outperforms conventional designs.

Key Results

  • Superior Early Decoding: In scenarios where only 15 out of 24 symbols were received, the Rateless AE (R-AE) maintained a usable error rate, while the conventional AE (C-AE) failed completely (BLER near 1.0).
  • The Trade-off: While the C-AE is slightly better when all 24 symbols are received (because it was specialized for exactly that length), the R-AE is vastly superior at all intermediate lengths (e.g., 15, 18, 21 symbols).
  • Power Constraint Sensitivity: The paper notes that using an average power constraint during training is vital for rateless performance. A fixed power constraint per symbol can actually inhibit the "rateless" property, as the decoder becomes too specialized to specific symbol energy levels.

Limitations and Future Work

  • Short Block Lengths: The study focuses on very short codes ($n=24$). Scaling autoencoders to very long block lengths remains a challenge due to the exponential growth of the message space and the complexity of the decoder.
  • Computational Complexity: While neural decoders are fast once trained, they are generally more computationally expensive than highly optimized classical decoders (like those for LDPC codes).
  • "Pseudo-Rateless": The authors acknowledge that their codes are not "strictly" rateless in the sense that they cannot generate an infinite stream of symbols (like a Fountain code); they are more accurately described as "rate-compatible" with a maximum length $n$. Extending this to truly infinite sequences is a suggested area for future research.

Summary updated

Abstract

Most of today's communication systems are designed to target reliable message recovery after receiving the entire encoded message (codeword). However, in many practical scenarios, the transmission process may be interrupted before receiving the complete codeword. This paper proposes a novel rateless autoencoder (AE)-based code design suitable for decoding the transmitted message before the noisy codeword is fully received. Using particular dropout strategies applied during the training process, rateless AE codes allow to trade off between decoding delay and reliability, providing a graceful improvement of the latter with each additionally received codeword symbol. The proposed rateless AEs significantly outperform the conventional AE designs for scenarios where it is desirable to trade off reliability for lower decoding delay.
BibTeX
@inproceedings{ninkovic2023rateless,
  author = {Vukan Ninkovic and Dejan Vukobratovic and Christian Häger and Henk Wymeersch and Alexandre Graell i Amat},
  title = {{Rateless Autoencoder Codes: Trading off Decoding Delay and Reliability}},
  booktitle = {ICC},
  year = {2023},
  doi = {10.1109/icc45041.2023.10278853},
}

Loading related content...

Figure