pCON: Polarimetric Coordinate Networks for Neural Scene Representations

Henry Peters, Yunhao Ba, Achuta KadambiCVPR 2023 · OpenReview

Abstract: Neural scene representations have achieved great success in parameterizing and reconstructing images, but current state of the art models are not optimized with the preservation of physical quantities in mind. While current architectures can reconstruct color images correctly, they create artifacts when trying to fit maps of polar quantities. We propose polarimetric coordinate networks (pCON), a new model architecture for neural scene representations aimed at preserving polarimetric information while accurately parameterizing the scene. Our model removes artifacts created by current coordinate network architectures when reconstructing three polarimetric quantities of interest. All code and data can be found at this link: http5://visual.ee.ucla.edu/pcon.htm.

The Challenge of Polarimetric Imaging

Neural scene representations—where a neural network (usually an MLP) maps coordinates $(x, y)$ to pixel values—have revolutionized 3D and 2D data storage and rendering. However, these models are typically optimized for visual fidelity in RGB space rather than the preservation of physical quantities.

When applied to polarimetric imaging, which measures the angle ($\phi$) and degree ($\rho$) of light polarization, standard models like SIREN and ACORN fail. Polarimetric maps often contain high spatial frequencies and sharp discontinuities, such as reflections on a checkerboard floor. SIREN tends to introduce high-frequency "salt and pepper" noise, while ACORN creates visible "tiling" or blocky artifacts at its internal grid boundaries. For downstream tasks like shape-from-polarization, these errors lead to massive inaccuracies in calculated surface normals.

Proposed Method: Polarimetric Coordinate Networks (pCON)

The core innovation of pCON is the integration of Singular Value Decomposition (SVD) with coordinate networks to separate learning tasks by frequency rather than spatial location.

The pCON architecture uses parallel network bands ($g_0, g_1, ... g_n$) to reconstruct polarimetric components through a coarse-to-fine summation process.
  1. SVD Decomposition: Target polarimetric maps are decomposed into singular values. Reconstructing the image involves summing rank-1 matrices.
  2. Band-wise Architecture: The model uses parallel MLPs, or "bands."
  3. Coarse-to-Fine Supervision: The first band learns low-frequency global structures using the largest singular values. Subsequent bands learn the "residuals"—the higher-frequency details represented by smaller singular values.
  4. Summation: The final output is the sum of all bands, avoiding tiling artifacts while maintaining sharp details.

Key Insights

  • SVD vs. Fourier: SVD proved more effective than Fourier transforms for coordinate networks as it avoids domain shifting and handles error propagation more gracefully.
  • Capacity Allocation: pCON focuses its neural complexity on volatile high-frequency areas rather than wasting energy on smooth regions.
  • Resolution Independence: As a continuous coordinate network, pCON can be queried at any resolution without upsampling artifacts.
  • Scalability: Accuracy can be traded for model size without retraining by simply dropping higher-frequency bands.

Results and Performance

pCON achieves significantly higher fidelity than existing baselines. Qualitative results demonstrate that pCON is uniquely capable of reconstructing complex polarization phenomena, such as those on a checkerboard floor, without introducing grid lines or grainy noise.

Qualitative and quantitative comparison on AoLP, DoLP, and unpolarized intensity ($I_{un}$) maps. pCON (Ours) achieves the highest SSIM/PSNR and eliminates artifacts seen in SIREN and ACORN.

Quantitatively, on Angle of Linear Polarization (AoLP) maps, pCON achieved an SSIM of 0.79, compared to SIREN's 0.59. Furthermore, the model offers a 7x reduction in storage (roughly 5 MB vs. 36 MB for raw data) with minimal loss in physical accuracy.

Limitations and Future Work

While pCON is a major step toward physically-aware neural representations, it currently focuses on 2D image fitting. Future work could extend this to 3D neural radiance fields (NeRFs). Additionally, the use of multiple parallel MLPs increases the total parameter count and training time compared to a single standard coordinate network.

Summary updated

Abstract

Neural scene representations have achieved great success in parameterizing and reconstructing images, but current state of the art models are not optimized with the preservation of physical quantities in mind. While current architectures can reconstruct color images correctly, they create artifacts when trying to fit maps of polar quantities. We propose polarimetric coordinate networks (pCON), a new model architecture for neural scene representations aimed at preserving polarimetric information while accurately parameterizing the scene. Our model removes artifacts created by current coordinate network architectures when reconstructing three polarimetric quantities of interest. All code and data can be found at this link: http5://visual.ee.ucla.edu/pcon.htm.
BibTeX
@inproceedings{peters2023pcon,
  author = {Henry Peters and Yunhao Ba and Achuta Kadambi},
  title = {{pCON: Polarimetric Coordinate Networks for Neural Scene Representations}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=sN28nCAxRZ},
}
Figure