HASwinNet: A Swin Transformer-Based Denoising Framework with Hybrid Attention for mmWave MIMO Systems

Xi Han, Houya Tu, Jiaxi Ying, Junqiao Chen, Zhiqiang XingEntropy 2026 · DOI

TL;DR

This paper introduces HASwinNet, a novel deep learning framework that leverages Swin Transformers and a dual-branch attention mechanism to significantly improve channel estimation and denoising in millimeter-wave (mmWave) massive MIMO systems. HASwinNet achieves superior accuracy and robustness, especially under challenging low-SNR and limited pilot conditions, making it a promising solution for future 6G backhaul and integrated sensing and communication (ISAC) applications.

What Problem Does This Paper Address?

Next-generation wireless networks, particularly 6G, are increasingly relying on millimeter-wave (mmWave) massive MIMO systems for high-capacity backhaul and integrated sensing and communication (ISAC). However, accurately estimating and denoising the communication channel in these systems is a significant challenge. This is due to several factors:

  • High Path Loss and Sparse Multipath: mmWave signals suffer from severe path loss and tend to have sparse, clustered multipath structures, making traditional estimation methods less effective.
  • Noise Sensitivity: The inherent noise in wireless channels, especially at low signal-to-noise ratios (SNRs), corrupts the channel estimates.
  • Limited Pilot Resources: To conserve precious bandwidth, systems often use a limited number of pilot symbols for channel estimation. This further exacerbates the accuracy problem.
  • Reconfigurable Intelligent Surfaces (RIS): While RIS enhances coverage, it adds complexity to channel estimation due to its passive nature and the need to estimate the combined channel.

Existing deep learning approaches, while promising, often struggle to simultaneously capture both the long-range angular dependencies and the fine-grained local spatial structures inherent in mmWave channels, especially under these challenging conditions.

Methods

HASwinNet tackles these challenges with a multi-stage, hybrid deep learning architecture:

  1. Hierarchical Swin Transformer Encoder: The framework begins with a Swin Transformer encoder. This architecture is well-suited for mmWave channels because its hierarchical, window-based attention mechanism efficiently balances local feature extraction with global context modeling, effectively capturing both spatial details and angular dependencies without the quadratic complexity of standard Transformers.
  2. Dual-Branch Processing: The encoded features are then fed into two parallel branches:
  3. Sparse Token Branch: This branch focuses on identifying and preserving the dominant multipath components. It uses Discrete Fourier Transform (DFT) to move to the angular domain, then applies a sparsity-aware gating mechanism (Top-K) to select the most significant angular components (sparse tokens). These tokens are then refined through self-attention and upsampling.
  4. Angular Refinement Branch: This branch enforces spectral consistency. It also uses DFT, followed by a Squeeze-and-Excitation (SE) module to adaptively re-weight spectral components, emphasizing informative angular bins and suppressing noise. An Inverse DFT (IDFT) maps these refined features back to the spatial domain.
  5. Decoder and Reconstruction: The outputs from both branches are fused in the latent domain using element-wise multiplication. This fused representation is then upsampled. To further stabilize training and preserve low-level signal details, a residual projection of the original noisy observation is added.
  6. Angular-Domain Perceptual Loss: A novel loss function is introduced, combining:
  7. NMSE Loss: Standard measure of reconstruction fidelity.
  8. Sparsity Regularization: Encourages the selection of dominant paths.
  9. Angular-Domain Perceptual Loss: Enforces spectral consistency in the angular domain, preserving the clustered multipath structure and improving generalization.

Datasets

The paper uses a simulated dataset based on the Saleh-Valenzuela (S-V) channel model. This model is a standard geometric channel model that effectively captures the sparse and directional characteristics of mmWave propagation.

  • Training Data: 50,000 samples.
  • Validation Data: 5,000 samples.
  • Testing Data: 5,000 samples.

The dataset includes noisy observations ($Y$) and corresponding ground-truth channel estimates ($H_{eq}$) generated under various Signal-to-Noise Ratio (SNR) conditions, ranging from -5 dB to 20 dB, and with different pilot overhead configurations (full pilot and half pilot).

Key Insights and Ideas

  • Decoupling for Clarity: The dual-branch approach is a key insight. By explicitly separating the tasks of identifying dominant sparse components (Sparse Token Branch) and refining spectral consistency (Angular Refinement Branch), the framework can more effectively exploit the distinct structural properties of mmWave channels.
  • Swin Transformer's Suitability: The paper highlights how the Swin Transformer's hierarchical, window-based attention is particularly well-suited for mmWave channels, which exhibit both local spatial features and long-range angular dependencies. It offers a computational advantage over standard Transformers for large channel dimensions.
  • Angular Domain as a Priori: The framework's strong reliance on the angular domain (via DFT) and the introduction of an angular-domain perceptual loss are crucial. This leverages the inherent angular sparsity of mmWave channels, which is a strong prior that can be exploited for more robust denoising, especially when pilot resources are scarce.
  • Hybrid Loss for Holistic Reconstruction: The composite loss function is more than just an aggregate; it's a carefully balanced set of objectives. The interplay between NMSE, sparsity, and angular consistency ensures that the reconstructed channel is not only accurate point-wise but also structurally coherent and representative of the underlying physical propagation.

Key Results

  • Superior NMSE and BER: HASwinNet consistently outperforms CNN, LSTM, and U-Net baselines in both Normalized Mean Squared Error (NMSE) and Bit Error Rate (BER) across the entire SNR range.
  • Low SNR Robustness: At low SNRs (-5 to 0 dB), HASwinNet shows a significant advantage, achieving NMSE values nearly an order of magnitude lower than CNN and LSTM at 20 dB.
  • Limited Pilot Performance: Even when using only half the pilot symbols, HASwinNet maintains a substantial performance margin over baselines, demonstrating its ability to effectively exploit angular sparsity and achieve higher pilot efficiency.
  • Hardware Imperfection Resilience: Experiments with discrete RIS phase shifts (2-bit quantization) show that HASwinNet's performance remains highly comparable to the ideal continuous phase case, indicating robustness to practical hardware limitations.
  • Computational Efficiency: HASwinNet has approximately 1.52 million learnable parameters and requires 0.84 GFLOPs, making it practical for real-time deployment in 6G backhaul.

Limitations and Future Work

  • Limitations:
    • The current work focuses on narrowband mmWave MIMO channels.
  • The performance under discrete phase shifts is evaluated for 1-bit and 2-bit quantization; higher levels of quantization or other hardware imperfections are not explored.
  • Future Work:
    • Extend HASwinNet to broadband and time-varying orthogonal frequency division multiplexing (OFDM) systems.
  • Conduct rigorous over-the-air (OTA) validation to further substantiate its applicability in realistic propagation conditions.
  • Explore its potential in ISAC scenarios, where accurate channel recovery can support high-resolution sensing capabilities.

Summary updated

Abstract

Millimeter-wave (mmWave) massive multiple-input, multiple-output (MIMO) systems are a cornerstone technology for integrated sensing and communication (ISAC) in sixth-generation (6G) mobile networks. These systems provide high-capacity backhaul while simultaneously enabling high-resolution environmental sensing. However, accurate channel estimation remains highly challenging due to intrinsic noise sensitivity and clustered sparse multipath structures. These challenges are particularly severe under limited pilot resources and low signal-to-noise ratio (SNR) conditions. To address these difficulties, this paper proposes HASwinNet, a deep learning (DL) framework designed for mmWave channel denoising. The framework integrates a hierarchical Swin Transformer encoder for structured representation learning. It further incorporates two complementary branches. The first branch performs sparse token extraction guided by angular-domain significance. The second branch focuses on angular-domain refinement by applying discrete Fourier transform (DFT), squeeze-and-excitation (SE), and inverse DFT (IDFT) operations. This generates a mask that highlights angularly coherent features. A decoder combines the outputs of both branches with a residual projection from the input to yield refined channel estimates. Additionally, we introduce an angular-domain perceptual loss during training. This enforces spectral consistency and preserves clustered multipath structures. Simulation results based on the Saleh-Valenzuela (S-V) channel model demonstrate that HASwinNet achieves significant improvements in normalized mean squared error (NMSE) and bit error rate (BER). It consistently outperforms convolutional neural network (CNN), long short-term memory (LSTM), and U-Net baselines. Furthermore, experiments with reduced pilot symbols confirm that HASwinNet effectively exploits angular sparsity. The model retains a consistent advantage over baselines even under pilot-limited conditions. These findings validate the scalability of HASwinNet for practical 6G mmWave backhaul applications. They also highlight its potential in ISAC scenarios where accurate channel recovery supports both communication and sensing.
BibTeX
@inproceedings{han2026haswinnet,
  author = {Xi Han and Houya Tu and Jiaxi Ying and Junqiao Chen and Zhiqiang Xing},
  title = {{HASwinNet: A Swin Transformer-Based Denoising Framework with Hybrid Attention for mmWave MIMO Systems}},
  booktitle = {Entropy},
  year = {2026},
  doi = {10.3390/e28010124},
}
Figure