Research progress of six degree of freedom (6DoF) video technology

Xu Wang, Qiong Liu, Zongju Peng, Junhui Hou, Hui Yuan, Tiesong Zhao, Yi Qin, Kejun Wu, Wenyu Liu, You YangJournal of Image and Graphics 2023 · DOI

Abstract

随着元宇宙概念的兴起,以6自由度(six degree of freedom, 6DoF)视频为代表的新一代交互式媒体技术得到产业界和学术界的广泛关注。6DoF视频隶属于多媒体通信领域,通过计算重构的方式向用户提供包括视角、光照、焦距和视场范围等多个维度的媒体交互与内容变化,能使千里之外的用户有身临其境、千人千面之感,与元宇宙具有的感知、计算、重构、协同和交互等技术特征具有高度重合性。因此,6DoF视频涵盖的技术体系可作为实现元宇宙的替代技术框架。本文提出了6DoF视频10个方面的40个问题,并将6DoF视频端到端技术链条归纳为生成、分发和呈现3个宏观阶段,随后围绕这3个技术阶段分别从内容采集与预处理、编码压缩与传输优化以及交互与呈现等方面阐述国内外研究进展。最后,本文围绕该领域当下的挑战,对未来趋势进行了讨论。

The Challenge of True Immersion

Traditional video is essentially a flat window. Even 360-degree "VR video" is usually limited to 3DoF (Three Degrees of Freedom), where you can rotate your head but cannot move your physical position without breaking the immersion. To achieve true presence, we need 6DoF, which adds translation (moving forward/backward, left/right, up/down).

However, 6DoF video requires an astronomical amount of data. Representing a 3D world from 2D camera sensors involves complex geometric reconstruction, massive bandwidth for transmission, and heavy computational loads for real-time rendering. This research provides a unified roadmap for researchers, categorizing the "generation, distribution, and visualization" phases and identifying 40 critical scientific challenges.

The 40-problem roadmap for 6DoF video systems, spanning generation, distribution, and visualization.

The End-to-End Pipeline

The 6DoF video ecosystem is divided into three macro-stages:

1. Generation: Acquisition & Pre-processing

Capturing 6DoF content requires sophisticated camera array geometries—parallel, diverging, converging, or surrounding. These systems use depth-sensing technologies like Active Structural Light, Time-of-Flight (ToF), and LiDAR to build a 3D representation. A significant shift is occurring from traditional geometric filtering to deep learning approaches (such as PointNet and GCNs) for "point cloud completion," where AI fills in data "holes" caused by occlusions.

A large-scale multi-camera array setup used for high-fidelity 6DoF video capture.

2. Distribution: Compression & Transmission

Standardization is critical for distribution. The field has evolved from Multi-view Video Coding (MVC) to advanced formats like MVD (Multiview Video plus Depth) and the latest MPEG standards: V-PCC (Video-based Point Cloud Compression) and G-PCC (Geometry-based). These codecs allow for massive bitrate reductions, making 3D streaming feasible over 5G/6G networks.

3. Visualization: Interaction & Display

The focus of visualization is Virtual View Synthesis. While Depth Image-Based Rendering (DIBR) remains a standard, there is explosive growth in Neural Radiance Fields (NeRF). NeRF uses 5D coordinates and neural networks to synthesize photorealistic views without explicit 3D geometry, allowing for higher visual quality by "learning" the light field of a scene.

  • The Metaverse Framework: 6DoF technology is the "alternative technical framework" for the Metaverse, sharing its core features of perception, computation, and interaction.
  • Human-Centric Optimization: Systems must account for human visual perception to prevent motion sickness, aiming for end-to-end latencies below 70ms.
  • Implicit Representations: The transition from explicit points/meshes to implicit neural representations (like NeRF) is a major milestone for visual fidelity.

Remaining Bottlenecks

Despite progress, several challenges remain. Current capture rigs are bulky and expensive; the future lies in lightweight, low-cost mobile arrays. Additionally, AI models must improve their generalization from synthetic training data to messy, real-world environments. Finally, optimizing the "Cloud-Edge-Terminal" synergy is essential to balance rendering power against battery life and latency.

Summary updated

Abstract

随着元宇宙概念的兴起,以6自由度(six degree of freedom, 6DoF)视频为代表的新一代交互式媒体技术得到产业界和学术界的广泛关注。6DoF视频隶属于多媒体通信领域,通过计算重构的方式向用户提供包括视角、光照、焦距和视场范围等多个维度的媒体交互与内容变化,能使千里之外的用户有身临其境、千人千面之感,与元宇宙具有的感知、计算、重构、协同和交互等技术特征具有高度重合性。因此,6DoF视频涵盖的技术体系可作为实现元宇宙的替代技术框架。本文提出了6DoF视频10个方面的40个问题,并将6DoF视频端到端技术链条归纳为生成、分发和呈现3个宏观阶段,随后围绕这3个技术阶段分别从内容采集与预处理、编码压缩与传输优化以及交互与呈现等方面阐述国内外研究进展。其中,在内容采集与预处理阶段,阐述了多视点联合采集、多视点与深度联合采集、深度图与点云预处理;在视频压缩与传输阶段,阐述了多视点视频编码、多视点+深度视频编码、光场图像压缩、焦栈图像压缩、点云编码压缩、6DoF视频传输优化;在交互与显示阶段,阐述了解码后滤波增强和虚拟视点合成。最后,本文围绕该领域当下的挑战,对未来趋势进行了讨论。
BibTeX
@article{wang2023research,
  author = {Xu Wang and Qiong Liu and Zongju Peng and Junhui Hou and Hui Yuan and Tiesong Zhao and Yi Qin and Kejun Wu and Wenyu Liu and You Yang},
  title = {{Research progress of six degree of freedom (6DoF) video technology}},
  journal = {Journal of Image and Graphics},
  year = {2023},
  doi = {10.11834/jig.230025},
}
Figure