Learning Channel Capacity With Neural Mutual Information Estimator Based on Message Importance Measure

Zhefan Li, Rui She, Pingyi Fan, Chenghui Peng, Khaled B. Letaief

TL;DR

This paper introduces a novel neural mutual information estimator called MMIE, which leverages the "Message Importance Measure" (MIM) to provide more stable and accurate channel capacity estimates than traditional methods. By embedding this estimator into a GAN-style cooperative framework, the authors demonstrate a system that simultaneously calculates the maximum data rate of a communication channel while designing the optimal codebook to achieve it.

What Problem Does This Paper Address?

In the quest for 6G and beyond, understanding channel capacity—the theoretical speed limit of a communication link—is vital. While we have elegant formulas for simple scenarios like Additive White Gaussian Noise (AWGN), real-world channels are often "black boxes" with complex, non-linear behaviors that lack closed-form solutions.

Existing approaches fall into two camps:

  1. Traditional Numerical Methods: These struggle with continuous inputs and high-dimensional data.
  2. Neural Estimators (e.g., MINE, iDIME): While powerful, these often suffer from high variance or "numerical explosions" (outputting infinity) when the Signal-to-Noise Ratio (SNR) is high, making them unreliable for practical engineering.

The authors aim to bridge this gap by creating an estimator that is both mathematically grounded and numerically robust across a wide range of channel conditions.

Methods

The core innovation is the use of the Message Importance Measure (MIM). Unlike standard Shannon entropy, which uses logarithmic functions, MIM uses exponential functions to characterize information. This change is significant because exponential functions are better at emphasizing "rare events" and provide better stability during neural network training.

The paper proposes two main components:

  • MMIE & $\alpha$-MMIE: A new mutual information estimator derived from the objective function of an MIM-based Generative Adversarial Network (GAN). The $\alpha$-parameterized version allows researchers to "center" the discriminator's output, preventing the vanishing gradient problems that plague other neural estimators at high SNRs.
  • Cooperative Framework (CORTICAL-based): They use a GAN-like architecture where a Generator acts as an encoder (creating the channel input/codebook) and a Discriminator acts as the MMIE (estimating the mutual information). They train together: the discriminator tries to accurately estimate the information flow, while the generator tries to maximize it.

Datasets

The authors evaluate their method using simulated communication environments rather than static datasets:

  • AWGN Channels: Tested in 2-dimensional and 10-dimensional configurations to represent varying levels of complexity.
  • Discrete Message Sets: A scenario with $M=8$ possible messages was used to compare the learned codebooks against standard industry benchmarks like 8-PSK (Phase Shift Keying).

Key Insights and Ideas

  • Exponential vs. Logarithmic: The shift from log-based loss (standard GANs) to exponential-based loss (MIM-GANs) provides a "smoother" landscape for the neural network to navigate. This leads to better resistance against "mode collapse" and more stable training.
  • Boundedness is Key: A major "aha" moment in the paper is the comparison of stability. Traditional neural estimators like iDIME can produce infinite values at high SNR because their underlying math involves dividing by zero or taking logs of zero. MMIE’s mathematical structure ensures its outputs remain bounded, leading to a 0% failure rate in tests where other models failed nearly 100% of the time.
  • Codebook "For Free": Because the framework is cooperative, once the system finishes estimating the capacity, the Generator has essentially "learned" the optimal way to shape signals for that specific channel, providing a ready-to-use codebook.

Key Results

  • Superior Stability: In 10-dimensional tests at 15dB SNR, the popular dDIME estimator failed 100% of the time due to numerical issues. MMIE and $\alpha$-MMIE maintained a 0% failure rate.
  • Higher Accuracy: MMIE consistently showed lower Root Mean Square Error (RMSE) and lower bias compared to the traditional KSG estimator and the well-known MINE (Mutual Information Neural Estimator).
  • Better Codebooks: The codebooks generated by the MMIE framework outperformed standard 8-PSK constellations, proving that the "learned" signal shapes were more efficient at navigating channel noise.

Limitations and Future Work

While MMIE solves many stability issues, it introduces a new hyperparameter, $\alpha$. Although the authors provide an empirical guideline for selecting $\alpha$ based on a rough pre-estimation of SNR, it still requires some manual tuning.

The paper primarily focuses on memoryless channels. Future work could involve extending this framework to channels with memory (where the current signal is affected by previous ones) or feedback loops, which are common in real-world mobile environments. Additionally, testing the framework on hardware-in-the-loop setups with actual radio hardware would be the next logical step toward industry adoption.

Summary updated

Abstract

Channel capacity estimation plays a crucial role in beyond 5G intelligent communications. Despite its significance, this task is challenging for a majority of channels, especially for the complex channels not modeled as the well-known typical ones. Recently, neural networks have been used in mutual information estimation and optimization. They are particularly considered as efficient tools for learning channel capacity. In this paper, we propose a cooperative framework to simultaneously estimate channel capacity and design the optimal codebook. First, we will leverage MIM-based GAN, a novel form of generative adversarial network (GAN) using message importance measure (MIM) as the information distance, into mutual information estimation, and develop a novel method, named MIM-based mutual information estimator (MMIE). Then, we design a generalized cooperative framework for channel capacity learning, in which a generator is regarded as an encoder producing the channel input, while a discriminator is the mutual information estimator that assesses the performance of the generator. Through the adversarial training, the generator automatically learns the optimal codebook and the discriminator estimates the channel capacity. Numerical experiments will demonstrate that compared with several conventional estimators, the MMIE achieves state-of-the-art performance in terms of accuracy and stability.
BibTeX
@article{li2024learning,
  author = {Zhefan Li and Rui She and Pingyi Fan and Chenghui Peng and Khaled B. Letaief},
  title = {{Learning Channel Capacity With Neural Mutual Information Estimator Based on Message Importance Measure}},
  journal = {IEEE Trans. Communications},
  year = {2024},
  url = {https://openreview.net/forum?id=B18I15y9IW},
}

Loading related content...

Figure