CNN-Based Signal Detection for Banded Linear Systems

Congmin Fan, Xiaojun Yuan, Ying-Jun Angela Zhang

TL;DR

This paper introduces a novel Convolutional Neural Network (CNN)-based signal detector designed specifically for banded linear systems, which are common in communication scenarios plagued by inter-symbol and inter-carrier interference. By utilizing a clever input preprocessing step to enforce shift-invariance, the proposed pure-convolutional network bypasses the "curse of dimensionality." The result is a high-accuracy, low-complexity detector that significantly outperforms traditional linear detectors and standard deep neural networks, while remaining completely robust to changes in system size without needing retraining.

What Problem Does This Paper Address?

In modern wireless communications, signals frequently interfere with one another. This manifests as Inter-Symbol Interference (ISI) in time-dispersive channels or Inter-Carrier Interference (ICI) in fast-fading channels. Mathematically, these scenarios are modeled as banded linear systems, where the channel matrix $\mathbf{H}$ has non-zero elements concentrated tightly around the main diagonal, as illustrated below:

A linear banded system with K = 5 and B = 1.

Traditional optimal detectors (like the BCJR algorithm) achieve maximum a posteriori probability (MAP) detection but suffer from exponential computational complexity relative to the channel bandwidth, making them unusable for large-scale systems. Linear alternatives like Least Squares (LS) or Linear Minimum Mean Square Error (LMMSE) are faster but leave a massive performance gap.

While deep learning offers a promising data-driven alternative, standard Multilayer Perceptrons (MLPs) suffer from the "curse of dimensionality"—their parameter counts explode as the system size grows, and they must be completely retrained whenever the system size changes.

Fig. 4. Architecture of a DNN-based detector.

Methods

The authors propose a specialized three-module architecture to bring the efficiency of CNNs to physical-layer signal detection:

  1. Input Preprocessing Module: CNNs inherently rely on shift-invariance (the idea that a feature is the same no matter where it appears). Raw channel matrices do not naturally possess this property. This module extracts and reshapes the complex non-zero channel coefficients and received signals into a real-valued vector $\mathbf{z}^{(0)}$ that artificially exhibits shift-invariance.
  2. CNN Module: A deep network consisting only of convolutional layers. Crucially, the authors omit pooling layers (which destroy fine-grained spatial features needed for symbol detection) and fully connected layers (which break scalability). The filter strides and sizes are mathematically tailored to span the local channel bandwidth ($2B+1$).
  3. Output Postprocessing Module: Maps the continuous sigmoid outputs from the network back to discrete transmitted symbols (e.g., BPSK $\pm 1$).

Beyond strictly banded systems, the authors extend their architecture to Near-Banded Systems:

  • Cyclic CNN (CCNN): Designed for 1-D near-banded systems like doubly selective OFDM, where interference wraps around the matrix corners (Figure 2).
  • 2-D CNN Detector: Tailored for Two-Dimensional Magnetic Recording (TDMR) systems experiencing two-dimensional inter-symbol interference (Figure 3).

Datasets

The approach was validated using extensive numerical simulations rather than static public datasets, allowing the authors to test generalization across various channel conditions:

  • Synthetic Banded Systems: Channels generated using Rayleigh fading (independent complex Gaussian coefficients) under both Additive White Gaussian Noise (AWGN) and non-Gaussian complex Gaussian mixture noise.
  • Practical System Models: Simulated environments including an Underwater Acoustic System ($K=1024$ subcarriers), a Doubly Selective OFDM system ($K=128$), and a TDMR system ($N=200, K=100$).

Key Insights and Ideas

  • Engineering Shift-Invariance: CNNs dominate computer vision because an object is an object no matter where it sits in an image. By restructuring the localized bands of a channel matrix into a structured vector, the authors successfully map physical radio interference to a pseudo-spatial domain, unlocking the true power of weight-sharing.
  • Ditching the Standard CNN Toolkit: Standard deep learning practices like max-pooling, dropout, and batch normalization actually degraded performance or added unnecessary complexity here. Preserving every single feature map output is vital when performing symbol-by-symbol classification.
  • Train Once, Deploy Anywhere: Because the network's parameters are tied strictly to the channel bandwidth ($B$) rather than the total system size ($K$), a model trained on a small system can be directly deployed on a massive system without retraining.

Key Results

  • Superior Accuracy: The CNN detector achieves more than an order of magnitude lower Bit Error Rate (BER) compared to classical LMMSE and LS detectors across various Signal-to-Noise Ratios (SNRs).
  • Flawless Generalization: A model trained on a small system size ($K_{train}=20$) performed identically well when tested on a much larger system ($K=100$), proving its immunity to system-size mismatches.
  • Noise Robustness: In environments with non-Gaussian mixture noise, the CNN detector maintained its performance edge, demonstrating that it learns the underlying channel structure rather than just fitting to Gaussian assumptions.

Limitations and Future Work

While highly effective, the current implementation primarily focuses on Binary Phase-Shift Keying (BPSK) modulation, though the authors note extensions to higher-order QAM are straightforward. Additionally, the framework assumes perfect Channel State Information (CSI) at the receiver.

For future work, the authors point toward integrating this detector into joint signal detection and channel decoding loops (e.g., passing soft decisions to/from an LDPC decoder). They also suggest that this localized CNN architecture could be generalized to solve other iterative network problems, such as resource management and power control.

Summary updated

Abstract

Banded linear systems arise in many communication scenarios, e.g., those involving inter-carrier interference and inter-symbol interference. Motivated by recent advances in deep learning, we propose to design a high-accuracy low-complexity signal detector for banded linear systems based on convolutional neural networks (CNNs). We develop a novel CNN-based detector by utilizing the banded structure of the channel matrix. Specifically, the proposed CNN-based detector consists of three modules: the input preprocessing module, the CNN module, and the output postprocessing module. With such an architecture, the proposed CNN-based detector is adaptive to different system sizes, and can overcome the curse of dimensionality, which is a ubiquitous challenge in deep learning. Through extensive numerical experiments, we demonstrate that the proposed CNN-based detector outperforms conventional deep neural networks and existing model-based detectors in both accuracy and computational time. Moreover, we show that CNN is flexible for systems with large sizes or wide bands. We also show that the proposed CNN-based detector can be easily extended to near-banded systems such as doubly selective orthogonal frequency division multiplexing (OFDM) systems and 2-D magnetic recording (TDMR) systems, in which the channel matrices do not have a strictly banded structure.
BibTeX
@article{fan2019cnn,
  author = {Congmin Fan and Xiaojun Yuan and Ying-Jun Angela Zhang},
  title = {{CNN-Based Signal Detection for Banded Linear Systems}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2019},
  doi = {10.1109/twc.2019.2924424},
}
Figure