Abstract
Multiview video can provide more immersive perception than traditional single 2-D video. It enables both interactive free navigation applications as well as high-end autostereoscopic displays on which multiple users can perceive genuine 3-D content without glasses. The multiview format also comprises much more visual information than classical 2-D or stereo 3-D content, which makes it possible to perform various interesting editing operations both on pixel-level and object-level. This survey provides a comprehensive review of existing multiview video synthesis and editing algorithms and applications. For each topic, the related technologies in classical 2-D image and video processing are reviewed. We then continue to the discussion of recent advanced techniques for multiview video virtual view synthesis and various interactive editing applications. Due to the ongoing progress on multiview video synthesis and editing, we can foresee more and more immersive 3-D video applications will appear in the future.
Introduction: The Multiview Challenge
The transition from traditional 2D video to immersive, multiview 3D experiences—such as 3D Television (3-DTV) and Free-viewpoint Television (FTV)—introduces significant technical hurdles. Capturing a scene from every possible angle is physically and financially impractical. To bridge this gap, researchers must address three primary challenges: color inconsistency across cameras, the "view gap" between physical camera positions, and the need for consistent 3D editing across multiple streams.
Technical Pillars of Multiview Processing
The survey categorizes the technical landscape into three main pillars:
1. Multiview Color Correction
Different cameras often capture colors differently due to varying angles and lighting conditions. This inconsistency ruins the 3D illusion and hinders data compression. Strategies range from pre-filtering and integrated encoding (using motion vectors to match colors) to post-processing. A notable technique is sparse Laplacian optimization, which corrects color profiles while preserving fine textures.
2. Virtual View Synthesis
The core method for generating views where no physical camera exists is Depth Image-Based Rendering (DIBR). This involves "warping" pixels from a known camera to a new position using depth maps.
A major challenge in synthesis is handling "disocclusions"—the holes left behind when foreground objects move. The paper discusses depth-aware inpainting, specifically patch-based algorithms that intelligently borrow background textures to fill these gaps without "bleeding" foreground colors.
3. Interactive Editing
Beyond simple pixel manipulation, the field is moving toward object-level editing. By using "3D proxies" (simple geometric shapes like cuboids), editors can rotate or scale objects in 3D space. The system then propagates these changes consistently across all video streams, maintaining the scene's 3D structure.
Key Insights and Results
- The MV+D Format: The Multiview-plus-Depth (MV+D) format is identified as the most practical representation, balancing 3D information with computational efficiency.
- Compression Benefits: Color correction is not just aesthetic; it significantly improves compression by making different views more predictable for encoders.
- Evaluation Metrics: In the absence of ground truth, "forward-reverse evaluation"—correcting a view and then attempting to undo the process—has become a standard for measuring quality.
- Maturity of Synthesis: DIBR has matured enough to support free navigation in sports broadcasts, although artifacts remain a challenge during rapid movement.
Future Directions
The survey highlights several remaining challenges:
- Large Baselines: Synthesis struggles when physical cameras are far apart, creating holes too large for current inpainting.
- Dynamic Backgrounds: Most models assume static backgrounds; handling moving elements like crowds or water remains difficult.
- Deep Learning Integration: The future lies in moving from purely geometric models to semantic systems that "understand" objects, allowing for more robust synthesis and intelligent editing.
- Real-time Performance: Many high-quality optimization methods are currently too slow for live 3D broadcasts.
Summary updated