TL;DR
This paper introduces a novel approach to 3D reconstruction using lensless cameras by combining coded illumination with a spatial baseline between the light source and the camera. This method significantly improves depth resolution and overall 3D scene quality compared to existing lensless imaging techniques, overcoming the inherent ill-conditioning of these systems.
What Problem Does This Paper Address?
Traditional cameras use lenses to focus light onto a sensor, enabling relatively straightforward 3D reconstruction. However, lensless cameras, which replace the lens with a coded mask, offer advantages like being flat, thin, and flexible, making them suitable for specialized applications. The major challenge with lensless cameras is achieving high-quality 3D reconstruction. This is because the sensor responses (point-spread functions or PSFs) for different depths are very similar, leading to an ill-conditioned inverse problem. Existing methods struggle with poor spatial and depth resolution. This paper aims to tackle this by enhancing the conditioning of the reconstruction problem.
Methods
The core of the proposed method involves two key innovations:
- Coded Illumination: Instead of relying on ambient light or a single uniform illumination, the system projects a sequence of distinct coded illumination patterns onto the scene. This active illumination strategy helps to disambiguate information from different parts of the scene.
- Baseline Separation: The illumination source (a projector) is intentionally placed at a distance (a baseline) from the lensless camera. This spatial separation is crucial. It causes depth-dependent shifts in the PSFs observed by the camera. Because these shifted PSFs are more distinct from each other than scaled versions (which are common in lensless imaging without a baseline), the system becomes better conditioned for 3D reconstruction.
The system models the lensless camera measurement as a linear function of the 3D scene intensity, where the PSF is dependent on depth. By capturing multiple measurements under different coded illuminations and with the baseline, the authors formulate a regularized inverse problem (specifically, using total variation regularization) to recover the 3D scene. They explore various types of binary illumination patterns, including uniform, random, shifting dots, and shifting lines, and evaluate their impact on reconstruction quality.
Datasets
The authors use the NYU depth dataset [38] for simulating their test scenes. For experimental validation, they use real 3D objects placed within a 40cm to 60cm depth range.
Key Insights and Ideas
- Baseline is Key for Depth Resolution: The spatial separation between the projector and camera is not just for illumination; it fundamentally alters how depth information is encoded in the sensor measurements. A larger baseline leads to more distinct depth-dependent PSF shifts, directly improving depth resolvability.
- Coded Illumination Enhances Conditioning: By using structured illumination patterns, the system gains more information per measurement, making the overall reconstruction problem better conditioned and less susceptible to noise and ill-posedness.
- Shifting Patterns Outperform Uniform: Among the tested illumination patterns, "shifting lines" and "shifting dots" consistently yielded better depth RMSE than uniform illumination, highlighting the benefit of spatially structured patterns.
- Trade-off Between Acquisition Time and Quality: Increasing the number of illumination patterns improves reconstruction quality but also increases the time required to capture all measurements, presenting a practical trade-off.
Key Results
- Simulation: Simulations showed that increasing the number of illumination patterns significantly reduces depth RMSE. "Shifting lines" and "shifting dots" patterns outperformed uniform illumination. Larger baselines (up to 7.5cm in simulation) also led to lower depth RMSE.
- Experiments: Real-world experiments with a prototype confirmed the simulation findings. The coded illumination approach, particularly with 48 shifting lines, produced significantly better spatial and depth resolution compared to uniform illumination. A 10.5cm baseline in experiments also demonstrated finer depth resolution than a 5.5cm baseline.
- Comparison to Pinhole: While an ideal pinhole camera with structured illumination performed better in simulations (due to superior conditioning), the proposed lensless approach offers a thin, lens-free alternative.
- Comparison to SweepCam: The proposed method showed better performance than a prior multi-shot lensless system (SweepCam) for objects at greater distances, where SweepCam struggled.
Limitations and Future Work
- Limitations:
- The current setup requires an additional illumination source, increasing cost and complexity.
- Capturing multiple shots increases acquisition time, limiting its use to static or slow-moving scenes.
- Future Work:
- Extending the method to dynamic scenes.
- Exploring new illumination patterns for even better 3D reconstruction.
- Co-designing illumination patterns, mask properties, and system arrangements for optimal performance.
- Improving the recovery algorithm with more sophisticated priors for 3D scenes.
Summary updated