Abstract
Glass segmentation in video remains a significant challenge due to the transparency and view-dependent reflections of glass surfaces. While polarization cues have improved static glass detection, their application to video is hindered by rapid state changes during camera movement. We present PGVS-Net, the first polarization-guided framework for video glass segmentation. By employing a multi-view spectral polarization propagation strategy and leveraging the new large-scale PGV-117 dataset, PGVS-Net achieves state-of-the-art temporal stability and accuracy, significantly outperforming existing RGB-only and single-frame polarization methods.
Introduction: The Challenge of Transparent Surfaces
Glass is often considered a "nightmare" material for computer vision. Its appearance is almost entirely defined by its surroundings and the current viewing angle, lacking intrinsic texture. Traditional RGB-based models frequently fail because glass can appear identical to the background. Recent advancements have utilized polarization—the orientation of light waves—to identify glass in static images. However, these methods struggle with video sequences where polarization states fluctuate as the camera moves, leading to inconsistent segmentation masks.
PGVS-Net addresses this by transitioning from single-frame analysis to multi-view spectral polarization propagation. By integrating information across multiple frames, the model gathers a more complete "polarimetric signature" of glass surfaces, compensating for views where polarization cues might be weak (e.g., near the Brewster angle).
Methods: The PGVS-Net Architecture
PGVS-Net treats video glass segmentation as a memory-based propagation task. It incorporates several key modules to handle multimodal data and temporal consistency:
- Polarization-Guided Integration (PGI): Fuses RGB and polarization features by using polarization cues to prioritize material-specific information.
- Spatio-Temporal Memory (STM) Variant: Maintains a memory bank of past frames, using a query-key mechanism to "remember" glass locations across different viewpoints.
- Cross Modal-Temporal Attention (CMTA): A transformer-style module that models short-term interactions between RGB, Angle of Linear Polarization (AoLP), and Degree of Linear Polarization (DoLP) across adjacent frames.
- Polarization Temporal Forward (PTF): Ensures spectral consistency by propagating features from the immediate previous frame to the current decoder, preventing boundary jitter.
The PGV-117 Dataset
To support this research, the authors introduced PGV-117, the first large-scale video dataset for polarization-guided glass segmentation. It contains 117 video sequences with 21,485 densely annotated RGB-P frames. The dataset covers diverse camera motions (circular, straight, up-down, etc.) and lighting conditions (low-light, HDR, saturated), providing a much-needed benchmark for the field.
Key Results and Performance
PGVS-Net sets a new performance standard on the PGV-117 test set. It achieved an Intersection over Union (IoU) of 84.60%, a substantial improvement over the best RGB-only video method (STCN, 80.85%) and the leading single-frame polarization method (PGSNet, 79.69%). The Balance Error Rate (BER) was also reduced to 10.02.
As shown in the results, PGVS-Net maintains a coherent and stable mask even as the camera rotates and lighting changes. The quantitative plot highlights the superior stability of PGVS-Net (blue line) compared to the significant performance fluctuations seen in previous models like PM-RCNN and PGSNet, which struggle as the polarization state (DoLP) varies.
Limitations and Future Work
PGVS-Net demonstrates that multi-view polarization propagation is a powerful tool for overcoming the visual ambiguity of glass in dynamic environments. While the model achieves high accuracy, it currently requires a ground-truth mask for the first frame and operates at 7 FPS. Future work will focus on optimizing inference speed and developing "zero-shot" initial mask estimation to enable fully autonomous operation.
Summary updated