Abstract
Depth maps are crucial for 3D video synthesis but are often captured at lower resolutions than texture data. This paper presents a novel up-sampling technique using fractal transforms. By leveraging the self-referential nature of depth data and extracting scale-invariant fractal parameters, the proposed method achieves resolution-independent up-sampling that preserves sharp edges more effectively than traditional interpolation or texture-guided methods.
Introduction: The Challenge of Depth Map Alignment
In modern 3D video systems, such as those using the Multi-View plus Depth (MVD) format, depth maps are indispensable for synthesizing virtual views. However, a significant hardware limitation exists: depth sensors typically capture data at much lower resolutions than high-definition texture cameras. To align these data sources, depth maps must be up-sampled.
Traditional methods like bilinear or bicubic interpolation often blur sharp edges, leading to geometry distortion and artifacts in the final 3D view. While "guided" filters attempt to use high-resolution texture images to sharpen depth maps, they are prone to failure if the texture and depth are misaligned. There is a clear need for a high-quality, edge-preserving up-sampling algorithm that is "resolution-independent"—capable of scaling to any arbitrary factor, not just fixed integers.
Methodology: Fractal-Based Reconstruction
The proposed approach utilizes Fractal Theory, specifically Iterated Function Systems (IFS), to treat the depth map as a self-referential structure. In this framework, small "range blocks" are described as transformed versions of larger "domain blocks."
The process involves three primary stages:
- Fractal Feature Extraction: The low-resolution depth map is partitioned into blocks. The algorithm identifies a contractive affine mapping—adjusting both geometry and luminance—that transforms a domain block to match a range block.
- Resolution Independence: These fractal parameters describe relationships rather than static pixels. Consequently, they lack an inherent scale, allowing the depth map to be reconstructed at any arbitrary resolution through iterative application of the transforms.
- Residual Enhancement: To account for approximation errors in fractal extraction, a residual is calculated (the difference between the original low-res map and its fractal reconstruction). This residual is up-sampled via standard interpolation and added back to the high-res fractal output to recover fine details.
Experimental Setup and Datasets
The method was evaluated using five standard 3D video test sequences, which are benchmarks in the coding community due to their inclusion of both texture and depth data:
- 1024x768 resolution: Book-Arrival, Balloons, Newspaper, and Kendo.
- 1920x1088 resolution: Dancer.
Key Results and Insights
The fractal approach treats image parameters as a set of instructions—a "DNA" of sorts—for building the image at any scale. Because depth maps consist largely of smooth regions separated by sharp geometric edges, fractals are mathematically better suited to represent them than natural textures.
Objective and Subjective Performance
The proposed method was compared against standard interpolation (Nearest, Bilinear, Bicubic) and advanced joint-filtering methods (JEDU and CDU).
- Quantitative Gains: For the Book-Arrival sequence, the method achieved a PSNR gain of 0.6dB to 0.82dB over the JEDU benchmark. In the Balloons sequence, it outperformed nearest-neighbor interpolation by over 4% in average PSNR.
- Visual Fidelity: Subjective comparisons revealed that the fractal method maintains significantly sharper object boundaries. Unlike competing methods, it avoids "staircase" artifacts and blurring at high zoom factors (2x, 4x, and 8x).
Limitations and Future Directions
The primary trade-off for this high-quality output is computational complexity. The search for optimal domain-range block matches is intensive, making the method approximately 0.11 to 0.21 seconds slower per frame than its competitors.
Future work will focus on optimizing the extraction process for speed. Additionally, while the current implementation primarily uses square partitions, the authors suggest that irregular partitioning—such as Triangle or Delaunay partitions—could further enhance edge accuracy by better capturing the underlying geometry of the depth map.
Summary updated