Polarization imaging captures a dimension of light that standard cameras completely ignore: the orientation of electromagnetic waves. By measuring how light vibrates after reflecting off surfaces, polarization sensors can distinguish camouflaged objects, separate man-made materials from natural backgrounds, and see through turbid water or heavy haze.
However, modern single-shot polarization cameras—such as Division-of-Focal-Plane (DoFP) sensors—rely on a mosaic array of micro-polarizers oriented at different angles (typically $0^\circ$, $45^\circ$, $90^\circ$, and $135^\circ$). This spatial sampling introduces a fundamental challenge: every pixel only measures a single polarization state. To recover the full polarization information, the missing states must be reconstructed through image processing pipelines like demosaicing, denoising, and deblurring.
For years, researchers attempted to treat polarization channels as if they were simply additional color channels. This approach fails. Recent research demonstrates that successful polarization image restoration requires physics-aware models that explicitly respect the mathematical constraints of polarized light and account for its spatial heterogeneity.
Why Traditional Image Pipelines Fail
To understand why standard computer vision algorithms fail on polarization data, we must look at how polarization information is calculated. The raw directional intensities ($I_0, I_{45}, I_{90}, I_{135}$) are combined to compute Stokes vectors, which are then used to derive the Degree of Linear Polarization (DoLP) and the Angle of Linear Polarization (AoLP).
$$\text{DoLP} = \frac{\sqrt{(I_0 - I_{90})^2 + (I_{45} - I_{135})^2}}{I_0 + I_{90}}$$
Because DoLP and AoLP are computed via non-linear combinations of multiple channels, any independent processing of individual channels breaks their underlying physical relationships.
As argued by Zhou et al., conventional deblurring algorithms do not account for these interdependencies. When a camera experiences motion blur, the blur affects all polarization angles simultaneously but differently depending on the scene's geometry. Applying a standard RGB deblurring algorithm in isolation corrupts the delicate ratios between channels, leading to severe degradation in downstream tasks like reflection removal.
A similar breakdown occurs with noise. In a benchmarking study on burst super-resolution, the authors demonstrate that sensor noise (shot and read noise) propagates non-linearly through the Stokes vector equations. Standard image denoising models assume uniform noise characteristics across channels. In polarization imaging, however, noise in a single directional frame severely distorts the final DoLP and AoLP calculations. Li et al. addressed this by designing "Pocoformer," a Transformer-based architecture that uses a hybrid attention mechanism specifically tailored to preserve these complex physical constraints during denoising.
Exploiting Inter-Channel Physical Constraints
If the strict mathematical relationships between polarization channels make independent processing dangerous, they also provide a powerful solution. These relationships can act as physical regularizers to guide the restoration process.
Consider the problem of demosaicing a DoFP sensor array. In standard color demosaicing, we assume that adjacent pixels share similar color distributions. In polarization demosaicing, we can leverage a fundamental physical law: the sum of intensities from any two orthogonal polarizers equals the total light intensity ($I_{\text{total}} = I_0 + I_{90} = I_{45} + I_{135}$).
DoFP Mosaic Array:
+----------+----------+
| I_0 | I_45 | --> Every pixel only measures ONE angle.
+----------+----------+ --> Physical Constraint:
| I_90 | I_135 | I_0 + I_90 == I_45 + I_135
+----------+----------+
Yan et al. operationalized this insight by introducing the Polarized Intensity Ratio Constraint (PIRC). Instead of interpolating missing pixel values in isolation, their method enforces the orthogonal intensity sum rule across the image while adjusting for real-world polarizer imperfections. By tying the channels together through this physical invariant, the model reconstructs fine spatial details without introducing the edge artifacts common to standard interpolation.
For single-chip sensors that capture both color and polarization simultaneously, the optimization space becomes even more complex. Wen et al. resolved this by using a sparse representation-based joint demosaicing framework. Their approach jointly optimizes color and polarization reconstruction but keeps their dictionary representations distinct, ensuring that color interpolation does not bleed into and corrupt the polarization signatures.
Handling Spatial Heterogeneity via Partitioning
In ideal laboratory settings, polarization properties might be uniform across a target. In the real world, polarization characteristics are highly heterogeneous. This is particularly true in challenging environments, such as underwater imaging, where light suffers from severe scattering and absorption.
Traditional polarization-based dehazing and underwater restoration methods rely on a global estimation strategy, assuming that background scattering and target polarization are uniform across the entire scene. However, as Cai et al. (2023) point out, different objects in an underwater scene exhibit vastly different polarization properties based on their material, distance, and orientation. A single, uniform estimation strategy inevitably fails, overcompensating in some regions and losing critical details in others.
Global Estimation (Fails):
[ Entire Image ] ---> Single Polarization Parameter ---> Artifacts & Blur
Partitioned Estimation (Succeeds):
/---> High-Polarization Region ---> Tailored Optimization
[ Input Image ] |
\---> Low-Polarization Region ---> Polarization Difference
To solve this, Cai et al. (2023) proposed a partition optimization method that segments the image into distinct regions based on the local degree of polarization. By breaking the scene down, the algorithm applies tailored optimization parameters to each region individually.
This concept was expanded by Cai et al. (2024) into a partition fusion framework. They noted that low-polarization regions are exceptionally difficult to restore because their signal mimics the unpolarized background scattering. By isolating these low-polarization zones, they could apply a specialized polarization difference method to extract the faint target signal, while treating high-polarization zones with standard inversion models. This spatial partitioning directly addresses scene heterogeneity, preventing the loss of critical information in low-contrast areas.
The Role of Hardware and Specialized Datasets
While algorithmic constraints and partitioning strategies are vital, the performance ceiling of these restoration methods is ultimately bounded by hardware capabilities and data availability.
On the hardware front, the ability to resolve distinct polarization states depends heavily on the Extinction Ratio (ER) of the sensor's micro-polarizers. If the ER is low, light from orthogonal angles leaks into the sensor, blurring the polarization states before any software processing even begins. Zhou et al. addressed this hardware bottleneck by developing a "super pixel" for long-wavelength infrared (LWIR) imaging using a plasmonic microcavity design. Achieving a record extinction ratio of over 100, this hardware breakthrough ensures that the raw data entering the image pipeline possesses highly distinct polarization states, making subsequent computational restoration significantly more effective.
On the data front, the rise of deep learning solutions (such as those proposed by Li et al. and Zhou et al.) has highlighted a critical shortage of specialized training data. Standard RGB datasets cannot be used to train polarization models because they lack the underlying physical properties like birefringence and angular constraints. Developing robust models requires the creation of dedicated datasets—such as PolarNS for noise modeling or specialized datasets capturing illumination variations—ensuring that neural networks learn the true physical laws of light propagation rather than superficial spatial correlations.
Key Foundational Work
High extinction ratio super pixel for long wavelength infrared polarization imaging detection based on plasmonic microcavity quantum well infrared photodetectors (2018) — contributed 1 concept elements to this direction.
A Sparse Representation Based Joint Demosaicing Method for Single-Chip Polarized Color Sensor (2021) — contributed 1 concept elements to this direction.