Abstract
Over the past decade, the utilization of UAVs has witnessed significant growth, owing to their agility, rapid deployment, and maneuverability. In particular, the use of UAV-mounted 360-degree cameras to capture omnidirectional videos has enabled truly immersive viewing experiences with up to 6DoF. However, achieving this immersive experience necessitates encoding omnidirectional videos in high resolution, leading to increased bitrates. Consequently, new challenges arise in terms of latency, throughput, perceived quality, and energy consumption for real-time streaming of such content. This paper presents a comprehensive survey of research efforts in UAV-based immersive video streaming, benchmarks popular video encoding schemes, and identifies open research challenges.
Introduction: The Challenge of Immersive Flight
Capturing 360-degree "omnidirectional" video (ODV) from a drone enables a "quasi-realistic" virtual environment, often referred to as 6-Degrees-of-Freedom (6DoF) viewing. However, the technical hurdles are massive. 360° video requires massive resolutions (4K to 8K) to look sharp in a headset, leading to high bitrates. Furthermore, Unmanned Aerial Vehicles (UAVs) are resource-constrained—they have limited battery life and onboard computing power. Finally, for applications like remote surveillance or "teleportation" (where a pilot flies via a VR headset), the end-to-end latency must be ultra-low (ideally <100ms). This paper bridges the gap between UAV communication and 360° video streaming by analyzing the entire "Glass-to-Glass" (G2G) pipeline.
System Architecture and Streaming Methods
The end-to-end pipeline for UAV-based immersive streaming involves complex stages from acquisition to rendering. The transmitter (UAV) must handle stitching, projection, encoding, and packaging before transmitting data over a network (e.g., 5G) to a receiver (VR headset).
Streaming strategies are generally categorized into viewport-independent and viewport-dependent methods. Viewport-dependent methods, such as tile-based or projection-based streaming, aim to reduce bandwidth by only sending high-quality data for the area the user is currently viewing.
Benchmarking and Latency Analysis
The authors evaluated five major video standards: AVC (H.264), HEVC (H.265), VVC (H.266), VP9, and AV1.
Key Findings:
- HEVC is the Current Winner: For real-time UAV use, HEVC hardware encoding (
hevc_nvenc) provides the best trade-off between coding efficiency and the low energy consumption required to preserve drone battery life. - AV1 for the Cloud: While AV1 software encoders (
libsvtav1) provide superior quality-per-bit, they are currently too slow for real-time onboard use and are better suited for cloud transcoding. - VVC’s Complexity: The newest standard, VVC (H.266), offers incredible compression but is currently unusable for live UAV streaming due to extreme encoding times (e.g., over an hour for a 10-second clip).
- The Latency Bottleneck: A critical finding from the 5G testbed is that the transmitter side (acquisition, stitching, and encoding) accounts for roughly 80% of the total G2G latency.
As shown in the table above, the total G2G latency reached 1745–1856 ms, which is significantly higher than the 100ms threshold required for seamless interactive control. Real-world tests also revealed that altitude increases interference from ground-based cell towers, further complicating throughput stability.
Conclusion and Future Directions
While 5G provides high throughput, network speed alone cannot solve the immersive streaming challenge. Future research must focus on reducing the 80% latency overhead at the transmitter. Potential solutions include moving toward "semantic communication"—conveying the meaning of a scene rather than raw bits—and leveraging Large Language Models (LLMs) to interpret 360° environments for UAV control. Additionally, multi-UAV swarms could collaboratively capture scenes to create "virtual MIMO" systems for even more immersive experiences.
The authors' new dataset of ten 360-degree aerial videos provides a vital resource for evaluating these future technologies under realistic flight conditions.
Summary updated