UAV Immersive Video Streaming: A Comprehensive Survey, Benchmarking, and Open Challenges

Mohit Kumar Sharma, Chen–Feng Liu, Ibrahim Farhat, Nassim Sehad, Wassim Hamidouche, Mérouane DebbaharXiv 2023 · DOI · arXiv

Abstract

Over the past decade, the utilization of UAVs has witnessed significant growth, owing to their agility, rapid deployment, and maneuverability. In particular, the use of UAV-mounted 360-degree cameras to capture omnidirectional videos has enabled truly immersive viewing experiences with up to 6DoF. However, achieving this immersive experience necessitates encoding omnidirectional videos in high resolution, leading to increased bitrates. Consequently, new challenges arise in terms of latency, throughput, perceived quality, and energy consumption for real-time streaming of such content. This paper presents a comprehensive survey of research efforts in UAV-based immersive video streaming, benchmarks popular video encoding schemes, and identifies open research challenges.

Introduction: The Challenge of Immersive Flight

Capturing 360-degree "omnidirectional" video (ODV) from a drone enables a "quasi-realistic" virtual environment, often referred to as 6-Degrees-of-Freedom (6DoF) viewing. However, the technical hurdles are massive. 360° video requires massive resolutions (4K to 8K) to look sharp in a headset, leading to high bitrates. Furthermore, Unmanned Aerial Vehicles (UAVs) are resource-constrained—they have limited battery life and onboard computing power. Finally, for applications like remote surveillance or "teleportation" (where a pilot flies via a VR headset), the end-to-end latency must be ultra-low (ideally <100ms). This paper bridges the gap between UAV communication and 360° video streaming by analyzing the entire "Glass-to-Glass" (G2G) pipeline.

System Architecture and Streaming Methods

The end-to-end pipeline for UAV-based immersive streaming involves complex stages from acquisition to rendering. The transmitter (UAV) must handle stitching, projection, encoding, and packaging before transmitting data over a network (e.g., 5G) to a receiver (VR headset).

End-to-end 360° video streaming pipeline for UAVs.

Streaming strategies are generally categorized into viewport-independent and viewport-dependent methods. Viewport-dependent methods, such as tile-based or projection-based streaming, aim to reduce bandwidth by only sending high-quality data for the area the user is currently viewing.

Benchmarking and Latency Analysis

The authors evaluated five major video standards: AVC (H.264), HEVC (H.265), VVC (H.266), VP9, and AV1.

Key Findings:

  • HEVC is the Current Winner: For real-time UAV use, HEVC hardware encoding (hevc_nvenc) provides the best trade-off between coding efficiency and the low energy consumption required to preserve drone battery life.
  • AV1 for the Cloud: While AV1 software encoders (libsvtav1) provide superior quality-per-bit, they are currently too slow for real-time onboard use and are better suited for cloud transcoding.
  • VVC’s Complexity: The newest standard, VVC (H.266), offers incredible compression but is currently unusable for live UAV streaming due to extreme encoding times (e.g., over an hour for a 10-second clip).
  • The Latency Bottleneck: A critical finding from the 5G testbed is that the transmitter side (acquisition, stitching, and encoding) accounts for roughly 80% of the total G2G latency.
Breakdown of Glass-to-Glass (G2G) latency in the experimental testbed.

As shown in the table above, the total G2G latency reached 1745–1856 ms, which is significantly higher than the 100ms threshold required for seamless interactive control. Real-world tests also revealed that altitude increases interference from ground-based cell towers, further complicating throughput stability.

Conclusion and Future Directions

While 5G provides high throughput, network speed alone cannot solve the immersive streaming challenge. Future research must focus on reducing the 80% latency overhead at the transmitter. Potential solutions include moving toward "semantic communication"—conveying the meaning of a scene rather than raw bits—and leveraging Large Language Models (LLMs) to interpret 360° environments for UAV control. Additionally, multi-UAV swarms could collaboratively capture scenes to create "virtual MIMO" systems for even more immersive experiences.

The authors' new dataset of ten 360-degree aerial videos provides a vital resource for evaluating these future technologies under realistic flight conditions.

Summary updated

Abstract

Over the past decade, the utilization of UAVs has witnessed significant growth, owing to their agility, rapid deployment, and maneuverability. In particular, the use of UAV-mounted 360-degree cameras to capture omnidirectional videos has enabled truly immersive viewing experiences with up to 6DoF. However, achieving this immersive experience necessitates encoding omnidirectional videos in high resolution, leading to increased bitrates. Consequently, new challenges arise in terms of latency, throughput, perceived quality, and energy consumption for real-time streaming of such content. This paper presents a comprehensive survey of research efforts in UAV-based immersive video streaming, benchmarks popular video encoding schemes, and identifies open research challenges. Initially, we review the literature on 360-degree video coding, packaging, and streaming, with a particular focus on standardization efforts to ensure interoperability of immersive video streaming devices and services. Subsequently, we provide a comprehensive review of research efforts focused on optimizing video streaming for timevarying UAV wireless channels. Additionally, we introduce a high resolution 360-degree video dataset captured from UAVs under different flying conditions. This dataset facilitates the evaluation of complexity and coding efficiency of software and hardware video encoders based on popular video coding standards and formats, including AVC/H.264, HEVC/H.265, VVC/H.266, VP9, and AV1. Our results demonstrate that HEVC achieves the best trade-off between coding efficiency and complexity through its hardware implementation, while AV1 format excels in coding efficiency through its software implementation, specifically using the libsvt-av1 encoder. Furthermore, we present a real testbed showcasing 360-degree video streaming over a UAV, enabling remote control of the drone via a 5G cellular network.
BibTeX
@misc{sharma2023uav,
  author = {Mohit Kumar Sharma and Chen–Feng Liu and Ibrahim Farhat and Nassim Sehad and Wassim Hamidouche and Mérouane Debbah},
  title = {{UAV Immersive Video Streaming: A Comprehensive Survey, Benchmarking, and Open Challenges}},
  year = {2023},
  eprint = {2311.00082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.00082},
}

Loading related content...

Figure