Abstract
Semantic segmentation has recently witnessed major progress, where fully convolutional neural networks (FCNs) have shown to perform well. However, most previous work focused on improving single image segmentation, often ignoring the rich temporal information available in video sequences. In this paper, we introduce a novel approach to implicitly utilize temporal data in videos for online semantic segmentation. The method relies on a fully convolutional network that is embedded into a gated recurrent architecture. This design receives a sequence of consecutive video frames and outputs the segmentation of the last frame. Convolutional gated recurrent networks are used for the recurrent part to preserve spatial connectivities in the image. Our proposed method, the Recurrent Fully Convolutional Network (RFCN), is tested for both binary and semantic video segmentation tasks across benchmarks including SegTrack V2, DAVIS, CityScapes, and Synthia. Results demonstrate that RFCN consistently improves upon baseline FCN performance by exploiting spatiotemporal information.
Introduction
While deep learning has revolutionized semantic segmentation—the task of assigning a category label to every pixel—most state-of-the-art models treat video as a collection of independent still images. This approach fails to capture temporal cues that are vital for disambiguating objects. For instance, a stationary object might be difficult to classify in a single frame, but its motion in a video sequence provides a clear signature for identification.
Previous attempts to integrate Recurrent Neural Networks (RNNs) into video processing often struggled with the "vectorization" problem. Standard RNNs require flattening 2D images into 1D vectors, which destroys spatial topology and leads to a parameter explosion. To address this, we propose an architecture that maintains spatial structure throughout the recurrent process.
The RFCN Architecture
The core innovation of the RFCN is the integration of spatial power from FCNs with the temporal memory of Gated Recurrent Units (GRUs) in a way that preserves spatial connectivity.
Convolutional Gated Recurrent Unit (Conv-GRU)
Unlike standard GRUs that use fully connected layers, the Conv-GRU replaces dot products with convolutional operations. This allows the hidden state to remain a 3D feature map rather than a flattened vector. By doing so, the network can learn temporal dependencies while respecting the local spatial relationships of pixels.
Sliding Window Processing
The model operates on a temporal window of consecutive frames. As shown in the architecture, the network processes these frames through the recurrent layers, accumulating information in the hidden state. The final output is a dense segmentation map for the most recent frame in the sequence. This design allows for both online (real-time) and batch processing.
Experimental Evaluation
We evaluated the RFCN across several diverse datasets to test its robustness in different environments:
- Binary Segmentation: On SegTrack V2 and DAVIS, the model focused on foreground-background separation.
- Semantic Segmentation: On CityScapes (urban street scenes) and Synthia (synthetic urban data), the model performed multi-class labeling.
Key Results
The RFCN consistently outperformed non-recurrent baselines across all metrics:
- Quantitative Gains: We observed a 5% and 3% improvement in F-measure for SegTrack V2 and DAVIS, respectively. In semantic tasks, mean IoU improved by 5.7% in Synthia and 3.5% in CityScapes.
- Motion Learning: Analysis showed that the recurrent units implicitly learn motion patterns. Accuracy gains were most significant for dynamic classes like pedestrians and vehicles, where temporal consistency is most beneficial.
Discussion and Conclusion
The RFCN serves as a "baseline booster." Rather than replacing existing FCN architectures, it provides a framework to wrap them in a recurrent structure, enabling them to exploit spatiotemporal data.
One current limitation is memory consumption; unrolling recurrent networks over many frames is GPU-intensive, which necessitated lower-resolution training for some datasets. Future work will explore more memory-efficient recurrent structures and the integration of explicit depth or shape constraints to further refine boundaries. Overall, the RFCN demonstrates that temporal information is a powerful, often underutilized resource for improving the accuracy and stability of video segmentation.
Summary updated