Abstract
The emerging field of semantic communication is driving research into end-to-end data transmission. By utilizing the powerful representation ability of deep learning models, learned data transmission schemes have exhibited superior performance compared to established source and channel coding methods. However, research efforts have mainly concentrated on architecture and model improvements toward a static target domain. Despite their successes, such learned models remain suboptimal due to limitations in model capacity and imperfect generalization, particularly when testing data distribution or channel response differs from training. To tackle this, we propose a novel online learned joint source and channel coding approach that leverages the deep learning model's overfitting property. Specifically, we update pre-trained models after deployment in a lightweight online fashion to adapt to distribution shifts in source data and environment domains. We take the overfitting concept to the extreme, proposing implementation-friendly methods to adapt the codec model or representations to individual data or channel state instances, leading to substantial gains in bandwidth ratio-distortion performance. Our experiments demonstrate that our approach outperforms existing state-of-the-art engineered transmission schemes (VVC combined with 5G LDPC) by up to 45% in bandwidth savings.
Introduction: The Amortization Gap
Traditional communication systems follow the Shannon paradigm, focusing on accurate bit transmission. Semantic Communication instead focuses on transmitting the meaning of data, often using Deep Joint Source-Channel Coding (JSCC). However, most JSCC models are "amortized"—trained on large datasets to perform well on average. This creates an amortization gap: a general model is rarely optimal for a specific, unique image or a rapidly changing wireless channel. When the test data distribution shifts or the signal-to-noise ratio (SNR) fluctuates, these static models become suboptimal.
Methodology: Online Learned NTSCC
The core innovation is Online Learned Nonlinear Transform Source-Channel Coding (NTSCC), which uses lightweight online optimization to bridge the amortization gap. The framework treats "overfitting"—usually considered a drawback in machine learning—as a tool for extreme compression. If a model is perfectly overfitted to a single image or channel state, it can represent that task with far fewer bits.
Adaptation Strategies
- Transmitter Adaptation (Tx-adapt): The transmitter performs gradient descent to optimize the encoder or latent representations for a specific frame. This improves feature compactness without requiring receiver changes.
- Transceiver Adaptation (TxRx-adapt): Both encoder and decoder are updated. This introduces a "model stream" alongside the standard "data stream." To minimize overhead, a Spike-and-Slab prior ensures only critical parameter updates are transmitted.
- Channel ModNet: A plug-in module for Vision Transformers (ViTs) that allows the model to adapt to instant Channel State Information (CSI) using SNR modulation, enabling agile resource allocation without gradient descent.
Experimental Results
The system was trained on the DIV2K dataset and evaluated using the Video Streaming Dataset 4K (VSD4K) across diverse categories like sports, dance, and gaming. Performance was tested over AWGN and COST2100 indoor fading channels.
- Bandwidth Efficiency: The TxRx-adapt scheme achieved up to 45% bandwidth savings compared to the VVC + 5G LDPC baseline for identical reconstruction quality.
- PSNR and Subjective Quality: The adaptive approach consistently outperformed static JSCC and traditional separation-based schemes. In a 2AFC user study, participants overwhelmingly preferred adaptive NTSCC images for their superior preservation of semantic details.
- Convergence: Approximately 50% of the adaptation gain is realized within the first minute of online training, making it practical for long-form video content.
Discussion and Conclusion
While the proposed framework offers significant efficiency gains, it introduces challenges regarding encoding latency and computational overhead. The transmitter requires high-end hardware (e.g., RTX 3090) for online learning, and frequent scene cuts in video can trigger constant re-adaptation. Future work will focus on "scene-aware" adaptation triggers and reducing the computational footprint for edge deployment. Nevertheless, this work represents a shift toward "software-defined" physical layers that evolve dynamically during a communication session.
Summary updated