Lightweight Neural Network with Knowledge Distillation for CSI Feedback

Yiming Cui, Jiajia Guo, Zheng Cao, Huaze Tang, Chao-Kai Wen, Shi Jin, Wang, Xin, Hou, XiaolinarXiv 2022 · DOI · arXiv

Abstract

Deep learning has shown promise in enhancing channel state information (CSI) feedback. However, many studies indicate that better feedback performance often accompanies higher computational complexity. Pursuing better performance-complexity tradeoffs is crucial to facilitate practical deployment, especially on computation-limited devices, which may have to use lightweight autoencoder with unfavorable performance. To achieve this goal, this paper introduces knowledge distillation (KD) to achieve better tradeoffs, where knowledge from a complicated teacher autoencoder is transferred to a lightweight student autoencoder for performance improvement. Specifically, two methods are proposed for implementation. Firstly, an autoencoder KD-based method is introduced by training a student autoencoder to mimic the reconstructed CSI of a pretrained teacher autoencoder. Secondly, an encoder KD-based method is proposed to reduce training overhead by performing KD only on the student encoder. Additionally, a variant of encoder KD is introduced to protect user equipment and base station vendor intellectual property. Numerical simulations demonstrate that the proposed methods can significantly improve the student autoencoder's performance, while reducing the number of floating point operations and inference time to 3.05%-5.28% and 13.80%-14.76% of the teacher network, respectively. Furthermore, the variant encoder KD method effectively enhances the student autoencoder's generalization capability across different scenarios, environments, and bandwidths.

Introduction

In Frequency Division Duplex (FDD) massive MIMO systems, the Base Station (BS) requires accurate Channel State Information (CSI) to perform effective beamforming. As the number of antennas grows, the overhead required to feed back this CSI from the User Equipment (UE) to the BS becomes a significant bottleneck. While Deep Learning (DL) based autoencoders have emerged as a powerful solution for compressing CSI into small "codewords," high-performance models are often computationally intensive. This poses a challenge for resource-constrained UEs, such as smartphones, where using a simple, lightweight model typically results in sacrificed feedback accuracy. This paper addresses this trade-off by applying Knowledge Distillation (KD) to CSI feedback, enabling a "tiny" student model to perform like a "giant" teacher model.

Proposed Architectures and Distillation Frameworks

The core methodology involves transferring "dark knowledge"—the subtle nuances of lossy compression—from a complex, pre-trained teacher network to a lightweight student network.

Network Architectures

The teacher network is designed with high complexity to ensure maximum reconstruction accuracy, while the student network is optimized for low computational overhead at the UE side.

Architectures of the teacher and student encoders alongside the common decoder, highlighting the complexity reduction in the student model.

As shown in Figure 2, the student encoder significantly reduces the number of layers and parameters compared to the teacher, targeting the strict power and processing constraints of mobile devices.

Distillation Methods

Three specific frameworks are introduced to implement the KD process:

  1. Autoencoder KD: The student is trained to match the "soft targets" (reconstructed outputs) of the teacher network, learning from the teacher's approximations rather than just the ground truth.
  2. Encoder KD: Distillation is focused solely on the encoder to reduce training time. The student mimics the teacher’s compressed codewords and utilizes the teacher's complex decoder at the BS during inference.
  3. Variant Encoder KD (IP-Protecting): This addresses industry concerns regarding proprietary architectures. It allows the BS and UE to train their respective parts independently by exchanging only CSI-codeword datasets, rather than sharing model weights.
Proposed knowledge distillation frameworks, including pretraining, encoder distillation, and the IP-protecting variant.

Performance Evaluation

The study utilizes datasets generated by the QuaDRiGa software, covering Urban Micro (UMi) and Indoor scenarios. Numerical simulations demonstrate that KD-trained student models significantly outperform those trained with standard "vanilla" methods.

For a compression ratio of 1/16, the Normalized Mean Square Error (NMSE) improved from -8.50 dB to -9.54 dB. More importantly, the student encoder requires only 3.05% to 5.28% of the teacher's FLOPs. Real-world testing on a Raspberry Pi 4 showed inference times under 1ms, representing a reduction to approximately 14% of the teacher's latency. Additionally, the KD process—particularly the Variant Encoder KD—acts as a regularizer, improving generalization across different channel environments (e.g., a 2.01 dB gain when moving from LOS to NLOS scenarios).

Conclusion

This paper demonstrates that Knowledge Distillation is a highly effective technique for achieving superior performance-complexity tradeoffs in CSI feedback. By aligning with 3GPP Release 18 discussions, these methods provide a practical path for deploying AI-enhanced feedback in 5G-Advanced and 6G systems. Future research will focus on incorporating temporal correlations and joint system-level optimization of the feedback module with beamforming.

Summary updated

Abstract

Deep learning has shown promise in enhancing channel state information (CSI) feedback. However, many studies indicate that better feedback performance often accompanies higher computational complexity. Pursuing better performance-complexity tradeoffs is crucial to facilitate practical deployment, especially on computation-limited devices, which may have to use lightweight autoencoder with unfavorable performance. To achieve this goal, this paper introduces knowledge distillation (KD) to achieve better tradeoffs, where knowledge from a complicated teacher autoencoder is transferred to a lightweight student autoencoder for performance improvement. Specifically, two methods are proposed for implementation. Firstly, an autoencoder KD-based method is introduced by training a student autoencoder to mimic the reconstructed CSI of a pretrained teacher autoencoder. Secondly, an encoder KD-based method is proposed to reduce training overhead by performing KD only on the student encoder. Additionally, a variant of encoder KD is introduced to protect user equipment and base station vendor intellectual property. Numerical simulations demonstrate that the proposed methods can significantly improve the student autoencoder's performance, while reducing the number of floating point operations and inference time to 3.05%-5.28% and 13.80%-14.76% of the teacher network, respectively. Furthermore, the variant encoder KD method effectively enhances the student autoencoder's generalization capability across different scenarios, environments, and bandwidths.
BibTeX
@misc{cui2022lightweight,
  author = {Yiming Cui and Jiajia Guo and Zheng Cao and Huaze Tang and Chao-Kai Wen and Shi Jin and Wang, Xin and Hou, Xiaolin},
  title = {{Lightweight Neural Network with Knowledge Distillation for CSI Feedback}},
  year = {2022},
  eprint = {2210.17113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.17113},
}
Figure