Combining Multi-level Contexts of Superpixel using Convolutional Neural Networks to perform Natural Scene Labeling

Aritra Das, Swarnendu Ghosh, Ritesh Sarkhel, Sandipan Choudhuri, Nibaran Das, Mita NasipuriarXiv 2018 · OpenReview

Abstract

Modern deep learning algorithms have triggered various image segmentation approaches. However, most of them deal with pixel-based segmentation, which is computationally expensive and often lacks sufficient local context. In this paper, we present a superpixel-level semantic segmentation approach that considers three different levels of neighborhood context. By training multiple Convolutional Neural Networks (CNNs) on these contexts and combining their predictions through ensemble methods—including max-voting, weighted-average, and the Dempster-Shafer theory of uncertainty—our method achieves superior performance on the ICCV09 dataset while maintaining real-time processing speeds.

Introduction: The Challenge of Pixel-Level Segmentation

Semantic image segmentation involves assigning a categorical label (e.g., "sky," "road," "tree") to every pixel in an image. While high accuracy is achievable with deep learning, two primary hurdles remain:

  1. Computational Efficiency: Processing millions of individual pixels is resource-intensive and often too slow for real-time applications.
  2. Contextual Ambiguity: A single pixel lacks the necessary information to distinguish between visually similar classes. For instance, a gray pixel could belong to a "building" or a "road" depending on its surroundings.

To address these issues, this research shifts the focus from individual pixels to "superpixels"—clusters of adjacent, similar pixels that provide inherent contextual information and reduce the total number of elements to be classified.

Methodology

The proposed pipeline involves superpixel generation, multi-level context extraction, parallel CNN processing, and ensemble-based decision making.

Overview of the proposed multi-level context CNN pipeline.

Superpixel Generation and Context Extraction

The system first applies the Simple Linear Iterative Clustering (SLIC) algorithm to group pixels into superpixels. To provide the network with varying degrees of spatial awareness, three versions of each superpixel patch are extracted:

  • 0N (Only superpixel patch): A raw 24x24 patch centered on the superpixel.
  • 1N (Patch + 1st Neighbor): A 32x32 patch including immediate neighbors.
  • 2N (Patch + 2nd Neighbor): A 48x48 patch including two layers of neighbors.
Visual representation of superpixel neighborhood levels (1N and 2N) used for context.

CNN Architecture

Three separate, shallow CNNs are trained, one for each context level. Each network architecture consists of two convolutional layers followed by max-pooling and a fully connected layer with 256 hidden units, ending in a Soft-Max layer for class probabilities.

Ensemble and Uncertainty Handling

To merge the outputs of the three CNNs, the researchers evaluated three strategies:

  1. Max-Voting: A simple majority rule based on the highest probability class from each network.
  2. Weighted Average: Assigns higher weight to the CNN that demonstrated superior performance during the training phase.
  3. Dempster-Shafer Theory: A mathematical framework used to handle uncertainty. By utilizing the confusion matrix from training, this method redistributes "belief" when networks are conflicted, helping to resolve ambiguities between similar classes like "building" and "object."

Experimental Results

The model was evaluated on the ICCV 09 Dataset, consisting of 715 natural scene images across 8 classes.

Performance and Benchmarking

The Weighted Average ensemble method achieved the highest classification accuracy at 77.14%. This approach outperformed several established baselines:

  • Baseline Pixel CRF: 74.3%
  • Farabet et al. (CNN + Superpixel): 74.56%
  • Tighe et al. (Features + Superpixel): 76.3%

The system excelled in identifying "Sky" (88.07%) and "Ground" (86.59%), though it struggled with the "Mountain" class (4% accuracy) due to a lack of representative training data in the dataset.

Real-Time Efficiency

By classifying superpixels rather than pixels, the system achieved a processing time of 25–30ms per image on a GTX 1080, making it suitable for real-time deployment.

Conclusion

This research demonstrates that combining multi-level neighborhood contexts significantly improves superpixel-level semantic labeling. The integration of Dempster-Shafer theory provides a robust mechanism for handling class confusion. Future work will focus on addressing data scarcity for rare classes and extending the framework to video segmentation to leverage temporal consistency.

Summary updated

Abstract

Modern deep learning algorithms have triggered various image segmentation approaches. However most of them deal with pixel based segmentation. However, superpixels provide a certain degree of contextual information while reducing computation cost. In our approach, we have performed superpixel level semantic segmentation considering 3 various levels as neighbours for semantic contexts. Furthermore, we have enlisted a number of ensemble approaches like max-voting and weighted-average. We have also used the Dempster-Shafer theory of uncertainty to analyze confusion among various classes. Our method has proved to be superior to a number of different modern approaches on the same dataset.
BibTeX
@misc{das2018combining,
  author = {Aritra Das and Swarnendu Ghosh and Ritesh Sarkhel and Sandipan Choudhuri and Nibaran Das and Mita Nasipuri},
  title = {{Combining Multi-level Contexts of Superpixel using Convolutional Neural Networks to perform Natural Scene Labeling}},
  year = {2018},
  eprint = {1803.05200},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.05200},
}
Figure