Abstract
The growing body of research on light fields underscores the increasing importance of 4D imaging. However, there remains a significant shortage of datasets specifically designed for light field video. This paper introduces a project aimed at filling this gap by developing synthetic light field video datasets using Blender. By providing high-quality sequences with ground-truth disparity maps, this work enables in-depth analysis of temporal and spatial redundancies, facilitating the development of future compression standards and 3D reconstruction techniques.
Introduction
Traditional 2D imaging captures light intensity at a single point on a sensor. In contrast, light fields capture both the intensity and the direction of light rays within a space. This 4D information enables advanced post-processing capabilities, such as digital refocusing and viewpoint shifting. While static light field imaging has matured, the transition to video—adding a temporal dimension—remains a significant challenge for the research community.
The Need for Synthetic Video Data
As international bodies like the Moving Picture Experts Group (MPEG) move toward standardizing light field video compression, the demand for high-quality datasets has surged. Capturing real-world light field video is prohibitively expensive and technically complex, often requiring large, synchronized camera arrays.
Furthermore, real-world data lacks "ground truth"—mathematically precise depth and disparity information. Synthetic environments solve this by providing controllable scenes where every pixel's 3D position is known, allowing for the rigorous validation of 3D reconstruction and disparity estimation algorithms.
Methodology
The researchers utilized Blender, an open-source 3D suite, integrated with Python scripts to simulate virtual camera arrays.
- Virtual Camera Rig: A multi-camera setup was implemented based on the 4D light field parametrization $L(u, v, s, t)$. The rig maintains parallel optical axes with precise offsets to minimize geometric distortion.
- Temporal Animation: Unlike static datasets, these scenes are animated, typically rendered at 20 frames per second (fps).
- Post-Processing: Raw renders were processed using ImageMagick and Matlab to generate standardized light field formats and high-precision disparity maps.
- Visualization: A modified light field viewer was developed to demonstrate automated focus-shifting over time, validating the dynamic behavior of the synthetic data.
Dataset Characteristics
The project introduced four distinct scenarios designed to stress-test various aspects of light field processing:
- Tower of Gods: Features objects at varying depths with moving elements (a boat and clouds), challenging disparity estimation during occlusions.
- Batman x Batman: Focuses on a ground plane with a wide range of disparities, extending from the immediate foreground to the far background.
- The Airplane: A detailed single-object scene used to evaluate how subtle disparity changes across a smooth surface impact compression efficiency.
- Return to Home: The most complex scenario, incorporating multiple objects with both positive and negative disparities and non-Lambertian (reflective) surfaces.
Key Insights and Results
The project successfully established several technical benchmarks:
- Disparity Ranges: Scenarios cover a broad depth spectrum, ranging from -6.6 to 1.1 in the "Return to Home" scene.
- Baseline Consistency: A consistent 50mm horizontal and vertical baseline was maintained between virtual cameras, ensuring data standardization.
- Temporal Challenges: The addition of movement highlighted the difficulty of handling "occlusions," where angular information changes abruptly as objects hide or reveal background elements.
- Precision Requirements: The data confirms that disparity is often fractional, necessitating high-precision estimators for AR/VR applications.
Conclusion and Future Work
While the current datasets provide a vital starting point, they are limited by short sequence lengths (typically 8 frames). Future efforts will focus on increasing frame counts to better test long-term temporal redundancy in compression algorithms. Additionally, the researchers plan to explore more advanced engines like Unity or Unreal Engine to achieve higher photorealism and more complex lighting effects, further pushing the boundaries of synthetic light field research.
Summary updated