YOLO-Based Semantic Communication With Generative AI-Aided Resource Allocation for Digital Twins Construction

Baoxia Du, Hongyang Du, Haifeng Liu, Dusit Niyato, Peng Xin, Jun Yu, Mingyang Qi, You Tang

This academic paper presents a novel framework for building Digital Twins (DTs) of agricultural environments—specifically apple orchards—using a combination of computer vision and generative AI. By shifting from traditional data transmission to "semantic communication," the authors demonstrate how to maintain high-fidelity digital replicas while drastically reducing the burden on wireless networks.

TL;DR

The researchers developed a semantic communication framework that uses a customized YOLOv7 model to extract only essential information (apples) from orchard images, reducing data transmission by 91%. They further optimized this process using a Diffusion Model—a type of generative AI—to intelligently allocate transmission power to the most critical data, ensuring high-quality Digital Twin updates even in noisy wireless environments.

What Problem Does This Paper Address?

Digital Twins require real-time synchronization between the physical world and a virtual model. In an agricultural setting, this often involves Unmanned Aerial Vehicles (UAVs) capturing high-resolution images and sending them to a server.

However, transmitting raw, high-definition video or images is incredibly "expensive" in terms of bandwidth and power, especially over 6G or edge networks. Much of this data is "noise"—leaves, branches, and sky that don't change the state of the Digital Twin. The challenge is two-fold:

  1. Efficiency: How do we transmit only the "meaning" (semantics) of the scene?
  2. Reliability: In a crowded wireless environment, how do we ensure the most important semantic information (e.g., a perfectly visible apple vs. one hidden by leaves) gets through clearly?

Methods

The authors propose a three-stage pipeline:

  1. Enhanced Object Detection (YOLOv7-HS): The authors modified the YOLOv7-X architecture to create "YOLOv7-HS." They introduced ELAN-H (Efficient Layer Aggregation Network-HorNet) to reduce computational complexity and the SimAM attention module. SimAM is a "parameter-free" 3D attention module that helps the model focus on small objects (like distant apples) without adding extra weight to the model, making it ideal for edge devices like UAVs.
  2. Semantic Extraction: Instead of sending the whole image, the UAV extracts "semantic features"—the cropped images of detected apples, their positions, and the detector's confidence scores.
  3. Generative AI-Aided Resource Allocation: This is the paper's most novel contribution. The authors use a Diffusion Model to solve the resource allocation problem. While diffusion models are famous for generating images (like Midjourney), here the model "generates" an optimal power allocation scheme. It treats the wireless environment as a state and learns to distribute transmission power among the detected objects to maximize a new metric called MIST (Metric for Image Semantic Transmission).

Datasets

The study utilizes the MinneApple dataset, a benchmark for apple detection and segmentation. It contains 670 labeled images and 331 unlabeled images featuring various apple varieties at different growth stages, characterized by high leaf occlusion and densely packed small objects.

Key Insights and Ideas

  • Confidence as a Proxy for Importance: The authors argue that an object detector's confidence score is a natural indicator of information value. High-confidence detections usually contain rich, clear features that are vital for the Digital Twin, while low-confidence ones are often obscured and less critical.
  • Semantic Compression: By transmitting only the detected objects and metadata, the system achieves a "lossy" compression that preserves 100% of the relevant information for the task while discarding 91% of the raw data.
  • Diffusion vs. Traditional RL: Using a diffusion-based approach for resource allocation allows for better exploration of complex "action spaces" (power levels) compared to standard Reinforcement Learning, leading to more robust performance as transmission distance increases.

Key Results

  • Data Reduction: The framework reduced the total data volume from 595.2 MB to 55.4 MB—a 91% decrease in communication costs.
  • Detection Accuracy: The YOLOv7-HS model achieved an [email protected] of 89.8%, outperforming the baseline YOLOv7-X (87.8%) while being significantly lighter (24% fewer parameters) and faster (34 FPS).
  • Transmission Quality: The diffusion-based resource allocation scheme consistently outperformed both "average allocation" and simple "confidence-based" schemes. It maintained higher MIST scores even at a transmission distance of 30 meters, where signal attenuation is high.

Limitations and Future Work

While the results are promising, a few areas remain for further exploration:

  • Hardware Constraints: Although the model is "lighter," running a modified YOLOv7 and a Diffusion-based optimizer on a small UAV in real-time still presents significant battery and thermal challenges.
  • Task Specificity: The current framework is highly specialized for apple orchards. Adapting this to more dynamic or heterogeneous environments (e.g., a construction site with many different types of objects) would require more complex semantic ontologies.
  • Multi-UAV Coordination: The paper focuses on a single UAV. Future work could look at how multiple edge devices might compete for or share resources in a shared agricultural metaverse.

Summary updated

Abstract

Digital Twins play a crucial role in bridging the physical and virtual worlds. Given the dynamic and evolving characteristics of the physical world, a huge volume of data transmission and exchange is necessary to attain synchronized updates in the virtual world. In this article, we propose a semantic communication framework based on you only look once (YOLO) to construct a virtual apple orchard with the aim of mitigating the costs associated with data transmission. Specifically, we first employ the YOLOv7-X object detector to extract semantic information from captured images of edge devices, thereby reducing the volume of transmitted data and saving transmission costs. Afterwards, we quantify the importance of each semantic information by the confidence generated through the object detector. Based on this, we propose two resource allocation schemes, i.e., the confidence-based scheme and the acrlong AI-generated scheme, aimed at enhancing the transmission quality of important semantic information. The proposed diffusion model generates an optimal allocation scheme that outperforms both the average allocation scheme and the confidence-based allocation scheme. Moreover, to obtain semantic information more effectively, we enhance the detection capability of the YOLOv7-X object detector by introducing new efficient layer aggregation network-horNet (ELAN-H) and SimAM attention modules, while reducing the model parameters and computational complexity, making it easier to run on edge devices with limited performance. The numerical results indicate that our proposed semantic communication framework and resource allocation schemes significantly reduce transmission costs while enhancing the transmission quality of important information in communication services.
BibTeX
@article{du2024yolo,
  author = {Baoxia Du and Hongyang Du and Haifeng Liu and Dusit Niyato and Peng Xin and Jun Yu and Mingyang Qi and You Tang},
  title = {{YOLO-Based Semantic Communication With Generative AI-Aided Resource Allocation for Digital Twins Construction}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  doi = {10.1109/jiot.2023.3317629},
}
Figure