Abstract
Immersive video serves as a key component of eXtended Reality (XR) that aims to create and interact with simulated virtual or hybrid environments. Such a technology allows users to experience immersive sensations that transcend time and space, and meanwhile continuously providing training data for emerging technologies like Embodied AI. Thanks to the advancements in sensing, computing, and display, recent years have witnessed many excellent works for XR and related hardware or software systems. However, challenges like high creation cost, lack of immersion, and limited scalability hinder the practical application of immersive video services. Whilst recently emerged generative artificial intelligence (GenAI) provides us with new insights in tackling existing challenges. In this paper, we conduct a comprehensive survey into the recent advances and future opportunities on how GenAI can benefit immersive video services. By introducing a systematic taxonomy, we meticulously classify the pertinent techniques and applications into three well-defined categories aligned with the pipeline of immersive video service: content creation, network delivery, and client-side display. This categorization enables a structured exploration of the diverse roles on how GenAI can benefit immersive video service, providing a framework for a more comprehensive understanding and evaluation of these technologies. To the best of our knowledge, this work is the first systematic survey of GenAI in XR settings, laying a foundation for future research in this interdisciplinary domain.
Introduction
Immersive video, encompassing 360° and volumetric content, is the backbone of eXtended Reality (XR). It enables users to experience virtual or hybrid environments that transcend physical limitations while providing critical training data for fields like Embodied AI. Despite its potential, the practical application of immersive video faces a "triple threat" of technical hurdles: high creation costs, massive network bandwidth requirements, and extreme computational loads for real-time rendering.
Generative Artificial Intelligence (GenAI) has emerged as a transformative solution to these challenges. This paper provides a systematic survey of how GenAI technologies—such as Diffusion models and Neural Radiance Fields (NeRF)—are being integrated into the immersive video pipeline to enhance scalability and performance.
The GenAI-Driven Immersive Video Pipeline
The integration of GenAI into immersive video can be categorized into three primary stages: content creation, network delivery, and client-side display.
1. Content Creation
Traditionally, high-quality 3D content required expensive LiDAR arrays and intensive manual labor. GenAI streamlines this through:
- Enhanced Capturing: Fusing multi-modal sensor data (RGB, LiDAR, audio) to create more accurate digital twins.
- Generative Creation: Utilizing Text-to-3D and Image-to-3D models (e.g., DreamTime, PC2) to democratize 3D design, allowing creators to build complex environments from simple text prompts.
2. Network Delivery
Volumetric data is notoriously massive. GenAI optimizes transmission via:
- Neural-enhanced Compression: Models like HiNeRV represent video as neural networks, often outperforming H.265 standards for multi-dimensional data.
- Semantic Communication (SemCom): Instead of treating every pixel equally, systems "understand" the scene, prioritizing bandwidth for focal areas (e.g., a person's face) while aggressively compressing static backgrounds.
3. Client-side Display
To enable high-fidelity experiences on consumer-grade hardware, GenAI focuses on upscaling and efficient rendering:
- Super-Resolution (SR): Upscaling low-resolution 3D models or 2D frames in real-time to maintain visual quality.
- Neural Rendering: Techniques like NeuVV use neural networks to simulate light and materials more efficiently than traditional rasterization.
Key Insights and Performance Gains
The survey highlights several breakthroughs in quantitative performance:
- Bandwidth Efficiency: The YuZu system demonstrated a 4.1x reduction in bandwidth usage by offloading heavy AI tasks to edge servers.
- Real-time Performance: YuZu also achieved 60 FPS on commodity hardware, reducing client-side rendering latency by 35%.
- Temporal Coherence: Advancements in temporal-consistent diffusion models have significantly reduced the "flickering" artifacts common in early AI-generated videos, ensuring smooth transitions between frames.
Future Frontiers and Challenges
While GenAI offers a path forward, several "open frontiers" remain:
- Latency vs. Model Size: Large Language Models (LLMs) currently introduce too much latency for real-time XR interactions.
- Privacy and Ethics: The collection of biometric data for immersive experiences raises concerns regarding deepfakes and unauthorized data synthesis.
- Scalability and Energy: Scaling GenAI to generate entire interactive cities in real-time while maintaining the battery life of mobile headsets remains a major engineering hurdle.
Conclusion
GenAI is not merely a buzzword but a necessary evolution for the XR industry. By automating creation, optimizing delivery, and enhancing rendering, GenAI lays the foundation for truly scalable and immersive virtual worlds.
Summary updated