Recursive Context Propagation Network for Semantic Scene Labeling

Abhishek Sharma, Oncel Tuzel, Ming-Yu LiuNeurIPS 2014 · OpenReview

Overview

The Recursive Context Propagation Network (rCPN) is a deep feed-forward architecture that achieves state-of-the-art semantic scene labeling by efficiently "sharing" information across an entire image. By combining a multi-scale Convolutional Neural Network (CNN) with a novel recursive structure, it outperforms complex graphical models while remaining orders of magnitude faster—capable of labeling a 256x256 image in just 0.07 seconds on a GPU.

The Challenge of Context in Scene Labeling

Semantic scene labeling requires assigning a category like "tree," "road," or "sky" to every pixel in an image. This is notoriously difficult because local appearance is often ambiguous. For example, a patch of green pixels could be grass, a bush, or a mossy rock. While humans are nearly perfect at this task because they naturally process the entire image for context, machines often struggle.

Historically, researchers used Conditional Random Fields (CRFs) or Structured SVMs to model these global relationships. However, these methods are often computationally expensive, slow during inference, and difficult to scale. The authors aim to bridge the performance gap between humans and machines by creating a fast, learnable neural network that propagates global context to every local region without the heavy computational overhead.

Proposed Methodology

The authors propose a two-part architecture to tackle this problem:

rCPN Architecture
  1. Local Feature Extraction: The model uses a multi-scale CNN to extract features at three different resolutions. This ensures the network captures both fine-grained details and a slightly broader field of view for every pixel.
  2. Recursive Context Propagation Network (rCPN): This is the core innovation. To avoid the massive computational cost of connecting every pixel to every other pixel, the image is grouped into "super-pixels" and organized into a binary parse tree.
  3. Bottom-Up Aggregation (Combiner): The network recursively merges features from child nodes to parent nodes until it reaches the "root," which represents a holistic description of the entire image.
  4. Top-Down Dissemination (Decombiner): The global information from the root is then passed back down the tree. Each local feature is updated with context from its parent, effectively "telling" each super-pixel what the rest of the image looks like.

Interestingly, the authors found that they don't need a perfect hierarchical segmentation. Using randomly generated trees based on spatial neighborhoods is sufficient to propagate context effectively.

Key Results and Performance

The model was evaluated on two major benchmarks: the Stanford Background dataset (715 images, 8 classes) and the more challenging SIFT Flow dataset (2,688 images, 33 classes).

Semantic Labeling Examples
  • Accuracy: On the SIFT Flow dataset, the rCPN achieved a Per Pixel Accuracy (PPA) of 79.6%, outperforming previous state-of-the-art methods.
  • Speed: The total time to label a 256x256 image is roughly 0.37 seconds (0.3s for super-pixel generation + 0.07s for the network pass). Previous methods often took several seconds or even minutes per image.
  • Ablation Insights: The authors demonstrated that the "Decombiner" (the top-down pass) is crucial. Without it, the model relies solely on local information and performs significantly worse.

Limitations and Future Work

While highly effective, the model has a few limitations. First, it currently acts as a "unary term"—it predicts the label for a super-pixel based on global context but doesn't explicitly model the relationship between adjacent labels in the final output (e.g., ensuring a "car" isn't floating in the "sky").

Additionally, the reliance on an off-the-shelf super-pixel algorithm means the model isn't entirely "end-to-end" in the modern sense, and the super-pixel generation itself acts as a speed bottleneck (taking up the majority of the 0.37s inference time). Future work will likely focus on scaling the architecture to even larger, more complex datasets like Microsoft COCO and fully integrating the segmentation pipeline.

Summary updated

Abstract

We propose a deep feed-forward neural network architecture for pixel-wise semantic scene labeling. It uses a novel recursive neural network architecture for context propagation, referred to as rCPN. It first maps the local visual features into a semantic space followed by a bottom-up aggregation of local information into a global representation of the entire image. Then a top-down propagation of the aggregated information takes place that enhances the contextual information of each local feature. Therefore, the information from every location in the image is propagated to every other location. Experimental results on Stanford background and SIFT Flow datasets show that the proposed method outperforms previous approaches. It is also orders of magnitude faster than previous methods and takes only 0.07 seconds on a GPU for pixel-wise labeling of a 256x256 image starting from raw RGB pixel values, given the super-pixel mask that takes an additional 0.3 seconds using an off-the-shelf implementation.
BibTeX
@inproceedings{sharma2014recursive,
  author = {Abhishek Sharma and Oncel Tuzel and Ming-Yu Liu},
  title = {{Recursive Context Propagation Network for Semantic Scene Labeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=rk-lpPbObr},
}
Figure