TL;DR
This paper introduces a novel sparse polarization sensor design and a corresponding deep learning architecture (SNA) to overcome the inherent trade-off between RGB image quality and polarization information resolution. By sparsely placing polarization filters and using a refined RGB image to guide the compensation of missing polarization data, they achieve significantly improved simultaneous acquisition of both modalities.
What Problem Does This Paper Address?
Traditional polarization sensors, while capable of capturing both RGB and polarization information in a single shot, suffer from a degradation in RGB image quality. This is because the polarizers on the sensor reduce light intensity, and to obtain a full RGB image, pixel binning (averaging) is often required, which further reduces spatial resolution. Conversely, reducing the number of polarization pixels to improve RGB quality leads to a lower resolution of polarization information. This creates a fundamental trade-off: better RGB means worse polarization, and vice-versa. The paper aims to break this trade-off, enabling high-quality acquisition of both simultaneously.
Methods
The core of the proposed solution lies in a two-pronged approach:
- Sparse Polarization Sensor: Instead of covering every pixel with a polarizer, the authors propose a sensor with sparsely arranged polarization filters (e.g., only 1/16th of pixels). This allows regular pixels to capture high-quality RGB data without the intensity reduction caused by polarizers.
- Stokes Network Architecture (SNA): This end-to-end deep learning framework is designed to compensate for the low-resolution polarization information. It consists of two main components:
- RGB Refinement Network (RGBRN): This network takes the raw RGB data from the sparse sensor and corrects for demosaicing artifacts and sparse pixel issues, producing a high-quality "refined" RGB image. This refined image serves as a crucial guide for the subsequent polarization compensation.
- Polarization Compensation Network (PCN): This network focuses on compensating the differential polarization components (S1 and S2) of the Stokes vector. It uses the refined RGB image and a mask indicating polarized pixels as input. The PCN employs a two-branch architecture: one for initial compensation and another for refinement, incorporating attention-based feature aggregation for better representation learning. The unpolarized component (S0) is derived from the refined RGB image.
Datasets
The authors utilized a combination of datasets for training and evaluation:
- Synthetic Dataset: A large-scale synthetic dataset was generated using Houdini, a 3D software, for procedural object and scene generation. This dataset is noted for its cost-effectiveness and diversity.
- Real-world Dataset: Real-world data was acquired using two methods:
- A polarization sensor (lower quality).
- Rotating a polarizer in front of a regular RGB camera (higher quality but more costly).
- Existing Datasets: The paper references and compares its approach against methods trained on existing polarization datasets like Ba [5], Lei [29], and Ono [42].
Key Insights and Ideas
- Breaking the Trade-off: The central idea is that a high-quality RGB image can effectively guide the reconstruction of sparse polarization information, thus resolving the long-standing trade-off.
- Leveraging RGB as a Guide: The refined RGB image is not just an output but a critical input for the polarization compensation network, providing spatial context and detail that sparse polarization data lacks.
- Focusing on Differential Components: Compensating only the differential polarization components (S1, S2) is more stable and efficient than trying to reconstruct all Stokes components from scratch, especially when dealing with sparse inputs.
- Attention for Feature Aggregation: The use of attention mechanisms in the PCN allows for more flexible and powerful learning of representations by selectively weighting global and visual features.
Key Results
- Improved Accuracy: The proposed SNA achieved significant improvements in both RGB and polarization metrics. Specifically, they demonstrated performance improvements of over 10 dB in PSNR for RGB and 1 dB for DoLP compared to conventional sensors.
- Quantitative Performance: In Table 3, their method (Ours) consistently outperformed baseline methods (Eqn. 2, Eqn. 3) across various sparsity ratios (r=1/4, 1/16, 1/64) for S0,1,2, S1,2, DoLP, and AoLP metrics. For instance, at r=1/16, their S1,2 RMSE was 4.151x10^-3, significantly lower than Eqn. 3's 4.353x10^-3.
- Qualitative Improvements: Figure 8 visually demonstrates the superior quality of RGB images and polarization information (DoLP and AoLP) obtained by their method compared to conventional sensors and other approaches.
- Noise Robustness: Table 7 shows that their method remains effective even under high noise conditions (Fn=3.6), producing high-quality RGB and polarization outputs with excellent balance.
- Efficiency: The processing speed was reported as 0.05 seconds per 768x576 image.
Limitations and Future Work
- Wavelength Information: The current approach uses a white color filter, limiting it to acquiring polarization information across the visible spectrum. Future work could explore acquiring wavelength-specific polarization data.
- Challenging Scenarios: Situations with imperfect correlation between RGB and polarization (e.g., photoelastic measurements of transparent objects) are noted as challenging for the current approach.
- Hardware Implementation: The network's large parameter count (184.8M) makes it unsuitable for direct hardware implementation. Future work will focus on making the network more compact.
- Prototyping: The authors intend to prototype their sparse polarization sensor to validate its practical application and further explore its incorporation into future cameras.
Summary updated