Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding

Jincheng Dai, Sixian Wang, Ke Yang, Kailin Tan, Xiaoqi Qin, Zhongwei Si, Kai Niu, Ping Zhang

Abstract

The emerging field of semantic communication is driving research into end-to-end data transmission. By utilizing the powerful representation ability of deep learning models, learned data transmission schemes have exhibited superior performance compared to established source and channel coding methods. However, research efforts have mainly concentrated on architecture and model improvements toward a static target domain. Despite their successes, such learned models remain suboptimal due to limitations in model capacity and imperfect generalization, particularly when testing data distribution or channel response differs from training. To tackle this, we propose a novel online learned joint source and channel coding approach that leverages the deep learning model's overfitting property. Specifically, we update pre-trained models after deployment in a lightweight online fashion to adapt to distribution shifts in source data and environment domains. We take the overfitting concept to the extreme, proposing implementation-friendly methods to adapt the codec model or representations to individual data or channel state instances, leading to substantial gains in bandwidth ratio-distortion performance. Our experiments demonstrate that our approach outperforms existing state-of-the-art engineered transmission schemes (VVC combined with 5G LDPC) by up to 45% in bandwidth savings.

Introduction: The Amortization Gap

Traditional communication systems follow the Shannon paradigm, focusing on accurate bit transmission. Semantic Communication instead focuses on transmitting the meaning of data, often using Deep Joint Source-Channel Coding (JSCC). However, most JSCC models are "amortized"—trained on large datasets to perform well on average. This creates an amortization gap: a general model is rarely optimal for a specific, unique image or a rapidly changing wireless channel. When the test data distribution shifts or the signal-to-noise ratio (SNR) fluctuates, these static models become suboptimal.

Conceptual overview of overfitting to specific source data and channel states to bridge the amortization gap.

Methodology: Online Learned NTSCC

The core innovation is Online Learned Nonlinear Transform Source-Channel Coding (NTSCC), which uses lightweight online optimization to bridge the amortization gap. The framework treats "overfitting"—usually considered a drawback in machine learning—as a tool for extreme compression. If a model is perfectly overfitted to a single image or channel state, it can represent that task with far fewer bits.

Adaptation Strategies

  1. Transmitter Adaptation (Tx-adapt): The transmitter performs gradient descent to optimize the encoder or latent representations for a specific frame. This improves feature compactness without requiring receiver changes.
  2. Transceiver Adaptation (TxRx-adapt): Both encoder and decoder are updated. This introduces a "model stream" alongside the standard "data stream." To minimize overhead, a Spike-and-Slab prior ensures only critical parameter updates are transmitted.
  3. Channel ModNet: A plug-in module for Vision Transformers (ViTs) that allows the model to adapt to instant Channel State Information (CSI) using SNR modulation, enabling agile resource allocation without gradient descent.
The TxRx-adapt mechanism showing the parallel Data Stream and Model Stream for transceiver updates.

Experimental Results

The system was trained on the DIV2K dataset and evaluated using the Video Streaming Dataset 4K (VSD4K) across diverse categories like sports, dance, and gaming. Performance was tested over AWGN and COST2100 indoor fading channels.

  • Bandwidth Efficiency: The TxRx-adapt scheme achieved up to 45% bandwidth savings compared to the VVC + 5G LDPC baseline for identical reconstruction quality.
  • PSNR and Subjective Quality: The adaptive approach consistently outperformed static JSCC and traditional separation-based schemes. In a 2AFC user study, participants overwhelmingly preferred adaptive NTSCC images for their superior preservation of semantic details.
  • Convergence: Approximately 50% of the adaptation gain is realized within the first minute of online training, making it practical for long-form video content.

Discussion and Conclusion

While the proposed framework offers significant efficiency gains, it introduces challenges regarding encoding latency and computational overhead. The transmitter requires high-end hardware (e.g., RTX 3090) for online learning, and frequent scene cuts in video can trigger constant re-adaptation. Future work will focus on "scene-aware" adaptation triggers and reducing the computational footprint for edge deployment. Nevertheless, this work represents a shift toward "software-defined" physical layers that evolve dynamically during a communication session.

Summary updated

Abstract

The emerging field semantic communication is driving the research of end-to-end data transmission. By utilizing the powerful representation ability of deep learning models, learned data transmission schemes have exhibited superior performance than the established source and channel coding methods. While, so far, research efforts mainly concentrated on architecture and model improvements toward a static target domain. Despite their successes, such learned models are still suboptimal due to the limitations in model capacity and imperfect optimization and generalization, particularly when the testing data distribution or channel response is different from that adopted for model training, as is likely to be the case in real-world. To tackle this, in this paper, we propose a novel online learned joint source and channel coding approach that leverages the deep learning model’s overfitting property. Specifically, we update the off-the-shelf pre-trained models after deployment in a lightweight online fashion to adapt to the distribution shifts in source data and environment domain. We take the overfitting concept to the extreme, proposing a series of implementation-friendly methods to adapt the codec model or representations to an individual data or channel state instance, which can further lead to substantial gains in terms of the end-to-end rate-distortion performance. Accordingly, the streaming ingredients include both the semantic representations of source data and the online updated decoder model parameters. The system design is formulated as a joint optimization problem whose goal is to minimize the loss function, a tripartite trade-off among the data stream bandwidth cost, model stream bandwidth cost, and end-to-end distortion. The proposed methods enable the communication-efficient adaptation for all parameters in the network without sacrificing decoding speed. Extensive experiments, including user study, on continually changing target source data and wireless channel environments, demonstrate the effectiveness and efficiency of our approach, on which we outperform existing state-of-the-art engineered transmission scheme (VVC combined with 5G LDPC coded transmission).
BibTeX
@article{dai2022toward,
  author = {Jincheng Dai and Sixian Wang and Ke Yang and Kailin Tan and Xiaoqi Qin and Zhongwei Si and Kai Niu and Ping Zhang},
  title = {{Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding}},
  journal = {IEEE J. Sel. Areas Commun.},
  year = {2022},
  doi = {10.48550/arxiv.2211.04339},
}
Figure