Introduction
Semantic Communication (SemCom) focuses on transmitting the "meaning" necessary for specific tasks, such as image classification, rather than raw data bits. While Large-scale AI (LAI) models excel at extracting this meaning, their massive size—often billions of parameters—makes them unsuitable for real-time deployment on resource-constrained mobile or IoT devices due to high latency and energy consumption. Furthermore, these models typically lack awareness of wireless channel conditions, leading to performance degradation under interference. This paper addresses these challenges by proposing a framework to distill LAI intelligence into lightweight, channel-aware systems.
Proposed Framework
The authors introduce the Lightweight Task-Oriented Semantic Communication (L-TOSC) framework, which leverages three primary innovations to bridge the gap between LAI performance and edge device constraints.
- Fast Distillation Method (FDM): To streamline training, the authors pre-compute and store only the "Top-K" most important outputs (logits) from a massive teacher model (e.g., CLIP-ViT-L/14). A logit smoothing technique then reconstructs a simplified version of the teacher’s knowledge. This approach reduces storage requirements by 8.7x and accelerates training by 2.4x.
- Information Bottleneck (IB) Loss: Grounded in IB theory, the framework employs a loss function that optimizes the trade-off between data compression (Rate) and task-relevant information retention (Distortion).
- Channel Adaptive Module: A lightweight module incorporates real-time Signal-to-Noise Ratio (SNR) feedback, allowing the transmitter to dynamically prioritize robust semantic features during noisy conditions.
Evaluation and Results
The framework was evaluated using the ImageNet-1k dataset, measuring Top-1 accuracy across various simulated wireless environments.
Key findings include:
- Extreme Compression and Low Latency: The distilled student model (L-TOSC-KD) is only 41.9 MB, representing just 3.6% of the teacher model's size. Inference latency was reduced from ~94ms to ~18ms, a 5x speedup.
- Robustness to Noise: At 0 dB SNR, the proposed method achieved a 6.6% higher accuracy than standard lightweight SemCom and a 12.4% lead over ResNet-based systems. Interestingly, the channel-aware student model can outperform its "channel-blind" teacher in low-SNR conditions.
- Learning Efficiency: Distillation from LAI models significantly boosts data efficiency; the model outperformed baselines even with a reduced training set of 0.6 million samples.
Conclusion
The L-TOSC framework successfully distills the capabilities of massive AI models into a format suitable for edge deployment. While a slight performance gap remains at very high SNRs compared to full-scale models, the gains in efficiency and noise robustness are substantial. Future research will explore extending this distillation approach to multi-modal tasks like Visual Question Answering and validating performance on physical IoT hardware.
Summary updated