TL;DR
This paper introduces the Recurrent Fully Convolutional Network (RFCN), a novel architecture that leverages temporal information in videos to improve semantic segmentation. By embedding a fully convolutional network within a Convolutional Gated Recurrent Unit (Conv-GRU), the model implicitly learns motion cues and spatial dependencies across frames, consistently outperforming static, frame-by-frame baselines.
What Problem Does This Paper Address?
While semantic segmentation—the task of assigning a category label to every pixel in an image—has seen massive gains thanks to Deep Learning, most models treat video as a mere collection of independent still images. This approach ignores a wealth of information: temporal cues.
In a single image, it might be hard to distinguish a stationary person from a statue, but in a video, the motion makes the distinction obvious. Previous attempts to use Recurrent Neural Networks (RNNs) for video struggled because standard RNNs require flattening images into 1D vectors. This "vectorization" destroys the spatial structure of the image and leads to an explosion in the number of parameters, making models nearly impossible to train on high-resolution data.
Methods
The authors propose the RFCN, which combines the spatial power of Fully Convolutional Networks (FCNs) with the temporal memory of Gated Recurrent Units (GRUs).
- Convolutional GRU (Conv-GRU): The key innovation is replacing the fully connected layers (dot products) inside a standard GRU with convolutional layers. This allows the "hidden state" of the network to remain a 3D feature map, preserving the spatial connectivity of pixels.
- Sliding Window Processing: The model takes a sequence of consecutive video frames as input. It processes these through the recurrent architecture and outputs a dense segmentation map for only the last frame in that window.
- Architecture Variants: The researchers tested several versions, including:
- RFC-Lenet/RFC-12s: Smaller models for baseline comparisons.
- RFC-VGG: Based on the VGG-F architecture, utilizing pre-trained weights to prevent overfitting.
- RFCN-8s: A high-performance version that uses "skip connections" to combine coarse, high-level features with fine, low-level details for precise boundaries.
Datasets
The model was evaluated across a diverse range of benchmarks to prove its versatility:
- Synthetic: Moving MNIST (handwritten digits moving in random directions) and Synthia (a large-scale synthetic dataset of urban scenes).
- Binary Video Segmentation: SegTrack V2 and DAVIS, which focus on separating a foreground object from the background.
- Real-world Semantic Segmentation: CityScapes (urban street scenes) and the Change Detection Dataset (indoor/outdoor moving object detection).
Key Insights and Ideas
- Motion as a Feature: The recurrent units don't just "smooth" the output; they implicitly learn motion patterns. This was evidenced by the fact that accuracy gains were highest for dynamic classes (like cars and pedestrians) compared to static classes (like buildings or sky).
- Efficiency through Convolutions: By using Conv-GRUs, the authors significantly reduced the parameter count compared to standard RNNs. This made it feasible to process larger images and deeper networks.
- The "Baseline Booster": The authors frame RFCN not as a replacement for existing FCNs, but as a method to improve them. Any FCN can essentially be "wrapped" in this recurrent structure to exploit spatiotemporal data.
Key Results
The RFCN consistently outperformed its non-recurrent counterparts across all tests:
- SegTrack V2 & DAVIS: Achieved a 5% and 3% improvement in F-measure, respectively.
- Synthia: Saw a 5.7% increase in mean Intersection over Union (mIoU).
- CityScapes: Improved categorical mIoU by 3.5%.
- Qualitative Improvement: Visual results showed that the RFCN produced much cleaner segmentations for moving objects, which are often blurred or misclassified by frame-by-frame models.
Limitations and Future Work
The primary limitation noted was memory consumption. Unrolling a recurrent network over multiple frames requires significant GPU memory, which forced the authors to use lower-resolution images for the CityScapes experiments. They noted that training on original resolutions would likely yield even better results.
For future work, the authors suggest incorporating explicit shape constraints or depth information (from RGB-D cameras) into the recurrent architecture to further refine the segmentation of complex objects.
Summary updated