Performance Analysis on Machine Learning-Based Channel Estimation

Kai Mei, Jun Liu, Xiaochen Zhang, Nandana Rajatheva, Jibo Wei

TL;DR

This paper provides the first theoretical analysis of the Mean Squared Error (MSE) performance of machine learning (ML)-based channel estimation. By employing hypothesis testing and developing a statistical model for linear ML modules, the authors derive a clear relationship between training data size and estimation performance, offering crucial insights for designing ML-based estimators, especially when training data is limited.

What Problem Does This Paper Address?

In wireless communication systems, accurately estimating the channel between the transmitter and receiver is crucial for reliable data recovery. Traditional methods like Least-Squares (LS) and Linear Minimum Mean Square Error (LMMSE) estimation have limitations: LS is sensitive to noise, while LMMSE requires knowledge of channel statistics and struggles with non-linear or non-stationary channels. Machine learning (ML), particularly deep learning (DL), has emerged as a promising alternative, demonstrating strong performance in simulations. However, a significant gap existed in understanding the theoretical performance of ML-based channel estimation, especially concerning its MSE and the impact of training data size. This paper aims to bridge this gap by providing a rigorous theoretical analysis of ML-based channel estimation's MSE performance.

Methods

The core of the paper's methodology lies in using hypothesis testing to analyze the MSE difference between an ML-based estimator and the optimal MMSE estimator. They define a scaled MSE difference, $\alpha = \Delta LE / LE1$, as the performance metric, where $\Delta LE$ is the MSE difference and $LE1$ is the MMSE.

For linear ML modules with low input dimensions, they develop a statistical model and derive an analytical relationship between the training data size (represented by $K=2M$, where $M$ is the number of training samples) and the scaled MSE difference $\alpha$. This analysis relies on two key assumptions:

  1. Independence of training losses: The training losses of the optimal and ML estimators are independent.
  2. Output error distribution: The output error of the ML estimator follows a complex Gaussian distribution.

These assumptions are argued to hold approximately when the learning module is linear and the input dimension is low. The paper also explores the performance of deep neural networks (DNNs), acknowledging that the assumptions might be violated in such cases, leading to different performance characteristics.

Datasets

The paper does not introduce new datasets. Instead, it relies on simulated data generated according to specified channel models (stationary and quasi-stationary) and system parameters (e.g., OFDM system configurations, power-delay profiles). The simulations are used to validate the theoretical analysis and evaluate the performance of ML-based estimators under various conditions.

Key Insights and Ideas

  • Theoretical Foundation for ML Channel Estimation: The paper provides the first theoretical framework for analyzing the MSE of ML-based channel estimation, moving beyond purely experimental validation.
  • MSE Upper Bound: Hypothesis testing reveals an upper bound on the MSE difference between ML and MMSE estimators, even for unseen data, offering theoretical support for the applicability of ML.
  • Data Size Matters (and is Predictable): A clear analytical relationship is established between the size of the training dataset and the estimation performance for linear ML modules. This allows for predicting performance and determining sufficient training data sizes.
  • Linear Modules are Data-Efficient (with caveats): For linear ML modules with low input dimensions, a relatively small training dataset (around 600 samples) can achieve performance close to the optimal MMSE.
  • DNNs are Data-Hungry: Deep learning models, while powerful, require significantly more training data to achieve comparable performance to linear models when data is limited, especially when their underlying assumptions are violated.
  • Input Dimension Trade-off: For linear models, there's a trade-off between input dimension and the required training data size. Higher input dimensions necessitate more data.

Key Results

  • Sufficient Data Size for Linear Models: The analysis suggests that for linear ML modules with low input dimensions, a training dataset size of approximately 600 samples is sufficient to achieve a scaled MSE difference ($\alpha$) below 0.1 (i.e., ML MSE is within 10% of MMSE).
  • Input Dimension Impact: Simulations show that the scaled MSE difference ($\alpha$) increases with the input dimension for linear models. For a training set of 600 samples, the input dimension can be up to 60 while maintaining $\alpha < 0.1$.
  • DNN Performance Degradation: With limited training data (600 samples), DNN-based channel estimation suffers significant performance degradation compared to LMMSE and even LS estimators, highlighting the need for much larger datasets for DL.
  • Subcarrier Partitioning for DNNs: For DNNs with limited data, partitioning OFDM symbols into smaller subsymbols to reduce the effective input dimension is shown to improve performance, with an input dimension of 60 offering a good tradeoff.

Limitations and Future Work

  • Assumptions for Linear Models: The analytical results for linear ML modules are derived under assumptions of independence and Gaussian output errors, which may not always hold perfectly in practice.
  • DNN Analysis: The theoretical analysis in Section III-B is primarily for linear models. While DNN performance is evaluated, a rigorous theoretical analysis for DNNs with violated assumptions is left as future work.
  • Future Work: The authors suggest extending the theoretical analysis to other learning structures like DNNs and investigating the statistical model of hypothesis testing for these complex models. Further theoretical work is also needed to verify the derived dataset sizes for high input dimensions.

Summary updated

Abstract

Recently, machine learning-based channel estimation has attracted much attention. The performance of machine learning-based estimation has been validated by simulation experiments. However, little attention has been paid to the theoretical performance analysis. In this paper, we investigate the mean square error (MSE) performance of machine learning-based estimation. Hypothesis testing is employed to analyze its MSE upper bound. Furthermore, we build a statistical model for hypothesis testing, which holds when the linear learning module with a low input dimension is used in machine learning-based channel estimation, and derive a clear analytical relation between the size of the training data and performance. Then, we simulate the machine learning-based channel estimation in orthogonal frequency division multiplexing (OFDM) systems to verify our analysis results. Finally, the design considerations for the situation where only limited training data is available are discussed. In this situation, our analysis results can be applied to assess the performance and support the design of machine learning-based channel estimation.
BibTeX
@article{mei2021performance,
  author = {Kai Mei and Jun Liu and Xiaochen Zhang and Nandana Rajatheva and Jibo Wei},
  title = {{Performance Analysis on Machine Learning-Based Channel Estimation}},
  journal = {IEEE Trans. Communications},
  year = {2021},
  doi = {10.1109/tcomm.2021.3083597},
}

Loading related content...

Figure