FAST: Flexible and Adaptive Semantic Transmission for Resource-constrained Multi-user Generative Semantic Communication

Yiru Wang, Wanting Yang, Zehui Xiong, Yuping Zhao, Shiwen Mao, Tony Q. S. Quek, H. Vincent PoorarXiv 2024 · DOI · arXiv

Abstract

The rapid advancement of generative artificial intelligence has spurred innovative approaches to semantic communication, giving rise to a new paradigm known as generative semantic communication (GSC). The integration of flexible cross-modal semantic extraction with generative capability-driven semantic inference substantially enhances semantic compression efficiency, demonstrating significant promise under communication resource constraints. Nonetheless, the stringent dependence on high computational power and the resulting latency continue to present major challenges, thereby limiting the feasibility of large-scale deployment. To address these challenges, we propose a novel GSC framework named FAST, which stands for flexible and adaptive semantic transmission. To accommodate limited computational resources, we propose a sequential semantic extraction method, where a temporal prompt engineering module orchestrates the distillation and transmission of key semantic units. Correspondingly, we introduce a sequential conditional denoising module at the receiver, which adapts the diffusion-based reconstruction to the progressively received input. To enhance overall task performance in multi-user semantic transmission, we propose a semantic-aware resource allocation method that optimizes bandwidth dynamically based on a joint consideration of semantic dependencies, user-level task priorities, and instantaneous channel conditions. Extensive experiments demonstrate that the proposed architecture achieves system precision comparable to conventional GSC systems while significantly reducing transmission latency and improving overall efficiency. These results confirm its enhanced potential for deployment in multi-user GSC scenarios with stringent communication and computational constraints.

Introduction

As we move toward the 6G era and massive IoT integration, traditional data transmission methods are increasingly hitting a "spectrum wall." Generative Semantic Communication (GSC) offers a promising solution: instead of transmitting raw pixels or high-bitrate data, the transmitter sends "semantic prompts" (such as text descriptions or edge maps) that a Generative AI (GAI) model at the receiver uses to reconstruct the original content.

While GSC is highly efficient at compression, it faces a significant bottleneck: latency. Conventional GSC systems typically follow an "all-or-nothing" approach, waiting for the entire semantic prompt to arrive before beginning the computationally intensive reconstruction process. In resource-constrained, multi-user environments—such as digital twin systems or autonomous robot fleets—this leads to substantial delays and inefficient use of edge computing resources. The FAST framework addresses this by transforming GSC into a "stream-and-refine" pipeline.

The FAST Framework

The FAST (Flexible and Adaptive Semantic Transmission) framework decomposes the transmission and reconstruction process into three coordinated modules designed to optimize both computational and communication resources.

The FAST framework architecture, illustrating the pipeline from multi-source data collection to diffusion-based reconstruction.

1. Temporal Prompt Engineering

Instead of extracting and sending all semantic units simultaneously, an RL agent (trained via Proximal Policy Optimization) determines the optimal sequence for extraction. By employing Invalid Action Masking, the agent efficiently learns which parts of a prompt are most critical for the final reconstruction, prioritizing high-impact semantic units to minimize initial latency.

2. Sequential Conditional Denoising

At the receiver, the system does not wait for the full prompt. It initiates the diffusion denoising process as soon as the first semantic units arrive. When subsequent information arrives mid-process, a Semantic Difference Calculation Subunit identifies how the noise space must be adjusted to incorporate new data without restarting the generation from scratch.

3. Semantic-aware Resource Allocation

In multi-user scenarios, bandwidth is not allocated solely based on Signal-to-Noise Ratio (SNR). The system dynamically shifts resources based on user task priority and the "semantic dependency" of the units currently being transmitted, ensuring that the most critical information across the network is prioritized.

Key Insights: Semantic Prioritization

A core finding of the FAST framework is the hierarchical importance of semantic types. Statistical analysis of the RL agent's behavior reveals a consistent prioritization of nouns (the foundation of the scene), followed by verbs (actions), and finally adjectives or styles (fine-grained details).

Comparison of semantic extraction sequences. Proper ordering (e.g., identifying the object before the action) is critical for accurate generative reconstruction.

As illustrated in the figure above, the order of extraction is vital. If an action (like "Jumping") is extracted before the subject ("Dog"), the generative model may fail to produce a coherent image. The FAST framework's ability to "steer" image generation mid-stream allows it to refine the output as specific semantic units arrive, effectively overlapping computation with transmission.

Experimental Evaluation

The framework was evaluated using the DIFFUSIONDB dataset, utilizing approximately 0.2 million prompts segmented into semantic units (NOUN, ADJECTIVE, STYLE, VERB, etc.) via the spaCy NLP library. Experiments were conducted using Stable Diffusion v1-4 on NVIDIA RTX 4090 GPUs.

Key performance metrics include:

  • Latency Reduction: FAST achieved a 21% reduction in residual system latency compared to baseline GSC methods.
  • Precision Gains: System precision (measured by CLIP score) improved by 29% over baselines lacking sequential denoising and semantic-aware allocation.
  • Efficiency: The overall system efficiency was approximately 1.6 times higher than conventional GSC architectures.

Conclusion

The FAST framework demonstrates that generative AI communication can be made practical for real-world, resource-constrained environments. By breaking the "wait-and-generate" cycle, FAST enables low-latency, high-precision semantic transmission. Future work will focus on expanding the framework to support diverse modalities such as video and 3D point clouds, as well as enhancing security against prompt-based vulnerabilities.

Summary updated

Abstract

The rapid advancement of generative artificial intelligence has spurred innovative approaches to semantic communication, giving rise to a new paradigm known as generative semantic communication (GSC). The integration of flexible cross-modal semantic extraction with generative capability-driven semantic inference substantially enhances semantic compression efficiency, demonstrating significant promise under communication resource constraints. Nonetheless, the stringent dependence on high computational power and the resulting latency continue to present major challenges, thereby limiting the feasibility of large-scale deployment. To address these challenges, we propose a novel GSC framework named FAST, which stands for flexible and adaptive semantic transmission. To accommodate limited computational resources, we propose a sequential semantic extraction method, where a temporal prompt engineering module orchestrates the distillation and transmission of key semantic units. Correspondingly, we introduce a sequential conditional denoising module at the receiver, which adapts the diffusion-based reconstruction to the progressively received input. To enhance overall task performance in multi-user semantic transmission, we propose a semantic-aware resource allocation method that optimizes bandwidth dynamically based on a joint consideration of semantic dependencies, user-level task priorities, and instantaneous channel conditions. Extensive experiments demonstrate that the proposed architecture achieves system precision comparable to conventional GSC systems while significantly reducing transmission latency and improving overall efficiency. These results confirm its enhanced potential for deployment in multi-user GSC scenarios with stringent communication and computational constraints.
BibTeX
@misc{wang2024fast,
  author = {Yiru Wang and Wanting Yang and Zehui Xiong and Yuping Zhao and Shiwen Mao and Tony Q. S. Quek and H. Vincent Poor},
  title = {{FAST: Flexible and Adaptive Semantic Transmission for Resource-constrained Multi-user Generative Semantic Communication}},
  year = {2024},
  eprint = {2407.15395},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.15395},
}
Figure