Multi-view Spectral Polarization Propagation for Video Glass Segmentation

Yu Qiao, Bo Dong, Ao Jin, Yu Fu, Seung-Hwan Baek, Felix Heide, Pieter Peers, Xiaopeng Wei, Xin Yang

Abstract

Glass segmentation in video remains a significant challenge due to the transparency and view-dependent reflections of glass surfaces. While polarization cues have improved static glass detection, their application to video is hindered by rapid state changes during camera movement. We present PGVS-Net, the first polarization-guided framework for video glass segmentation. By employing a multi-view spectral polarization propagation strategy and leveraging the new large-scale PGV-117 dataset, PGVS-Net achieves state-of-the-art temporal stability and accuracy, significantly outperforming existing RGB-only and single-frame polarization methods.

Introduction: The Challenge of Transparent Surfaces

Glass is often considered a "nightmare" material for computer vision. Its appearance is almost entirely defined by its surroundings and the current viewing angle, lacking intrinsic texture. Traditional RGB-based models frequently fail because glass can appear identical to the background. Recent advancements have utilized polarization—the orientation of light waves—to identify glass in static images. However, these methods struggle with video sequences where polarization states fluctuate as the camera moves, leading to inconsistent segmentation masks.

PGVS-Net addresses this by transitioning from single-frame analysis to multi-view spectral polarization propagation. By integrating information across multiple frames, the model gathers a more complete "polarimetric signature" of glass surfaces, compensating for views where polarization cues might be weak (e.g., near the Brewster angle).

Methods: The PGVS-Net Architecture

PGVS-Net treats video glass segmentation as a memory-based propagation task. It incorporates several key modules to handle multimodal data and temporal consistency:

  • Polarization-Guided Integration (PGI): Fuses RGB and polarization features by using polarization cues to prioritize material-specific information.
  • Spatio-Temporal Memory (STM) Variant: Maintains a memory bank of past frames, using a query-key mechanism to "remember" glass locations across different viewpoints.
  • Cross Modal-Temporal Attention (CMTA): A transformer-style module that models short-term interactions between RGB, Angle of Linear Polarization (AoLP), and Degree of Linear Polarization (DoLP) across adjacent frames.
  • Polarization Temporal Forward (PTF): Ensures spectral consistency by propagating features from the immediate previous frame to the current decoder, preventing boundary jitter.
Visualization of RGB, AoLP, and DoLP inputs alongside segmentation results for different frames.

The PGV-117 Dataset

To support this research, the authors introduced PGV-117, the first large-scale video dataset for polarization-guided glass segmentation. It contains 117 video sequences with 21,485 densely annotated RGB-P frames. The dataset covers diverse camera motions (circular, straight, up-down, etc.) and lighting conditions (low-light, HDR, saturated), providing a much-needed benchmark for the field.

Key Results and Performance

PGVS-Net sets a new performance standard on the PGV-117 test set. It achieved an Intersection over Union (IoU) of 84.60%, a substantial improvement over the best RGB-only video method (STCN, 80.85%) and the leading single-frame polarization method (PGSNet, 79.69%). The Balance Error Rate (BER) was also reduced to 10.02.

Quantitative comparison of mIoU stability over frame indices for different methods.

As shown in the results, PGVS-Net maintains a coherent and stable mask even as the camera rotates and lighting changes. The quantitative plot highlights the superior stability of PGVS-Net (blue line) compared to the significant performance fluctuations seen in previous models like PM-RCNN and PGSNet, which struggle as the polarization state (DoLP) varies.

Limitations and Future Work

PGVS-Net demonstrates that multi-view polarization propagation is a powerful tool for overcoming the visual ambiguity of glass in dynamic environments. While the model achieves high accuracy, it currently requires a ground-truth mask for the first frame and operates at 7 FPS. Future work will focus on optimizing inference speed and developing "zero-shot" initial mask estimation to enable fully autonomous operation.

Summary updated

Abstract

In this paper, we present the first polarization-guided video glass segmentation propagation solution (PGVS-Net) that can robustly and coherently propagate glass segmentation in RGB-P video sequences. By leveraging spatiotemporal polarization and color information, our method combines multi-view polarization cues and thus can alleviate the view dependence of single-input intensity variations on glass objects. We demonstrate that our model can outperform glass segmentation on RGB-only video sequences as well as produce more robust segmentation than per-frame RGB-P single-image segmentation methods. To train and validate PGVS-Net, we introduce a novel RGB-P Glass Video dataset (PGV-117) containing 117 video sequences of scenes captured with different types of camera paths, lighting conditions, dynamics, and glass types.
BibTeX
@inproceedings{qiao2023multi,
  author = {Yu Qiao and Bo Dong and Ao Jin and Yu Fu and Seung-Hwan Baek and Felix Heide and Pieter Peers and Xiaopeng Wei and Xin Yang},
  title = {{Multi-view Spectral Polarization Propagation for Video Glass Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.02122},
}
Figure