Abstract
Modern deep learning algorithms have triggered various image segmentation approaches. However, most of them deal with pixel-based segmentation, which is computationally expensive and often lacks sufficient local context. In this paper, we present a superpixel-level semantic segmentation approach that considers three different levels of neighborhood context. By training multiple Convolutional Neural Networks (CNNs) on these contexts and combining their predictions through ensemble methods—including max-voting, weighted-average, and the Dempster-Shafer theory of uncertainty—our method achieves superior performance on the ICCV09 dataset while maintaining real-time processing speeds.
Introduction: The Challenge of Pixel-Level Segmentation
Semantic image segmentation involves assigning a categorical label (e.g., "sky," "road," "tree") to every pixel in an image. While high accuracy is achievable with deep learning, two primary hurdles remain:
- Computational Efficiency: Processing millions of individual pixels is resource-intensive and often too slow for real-time applications.
- Contextual Ambiguity: A single pixel lacks the necessary information to distinguish between visually similar classes. For instance, a gray pixel could belong to a "building" or a "road" depending on its surroundings.
To address these issues, this research shifts the focus from individual pixels to "superpixels"—clusters of adjacent, similar pixels that provide inherent contextual information and reduce the total number of elements to be classified.
Methodology
The proposed pipeline involves superpixel generation, multi-level context extraction, parallel CNN processing, and ensemble-based decision making.
Superpixel Generation and Context Extraction
The system first applies the Simple Linear Iterative Clustering (SLIC) algorithm to group pixels into superpixels. To provide the network with varying degrees of spatial awareness, three versions of each superpixel patch are extracted:
- 0N (Only superpixel patch): A raw 24x24 patch centered on the superpixel.
- 1N (Patch + 1st Neighbor): A 32x32 patch including immediate neighbors.
- 2N (Patch + 2nd Neighbor): A 48x48 patch including two layers of neighbors.
CNN Architecture
Three separate, shallow CNNs are trained, one for each context level. Each network architecture consists of two convolutional layers followed by max-pooling and a fully connected layer with 256 hidden units, ending in a Soft-Max layer for class probabilities.
Ensemble and Uncertainty Handling
To merge the outputs of the three CNNs, the researchers evaluated three strategies:
- Max-Voting: A simple majority rule based on the highest probability class from each network.
- Weighted Average: Assigns higher weight to the CNN that demonstrated superior performance during the training phase.
- Dempster-Shafer Theory: A mathematical framework used to handle uncertainty. By utilizing the confusion matrix from training, this method redistributes "belief" when networks are conflicted, helping to resolve ambiguities between similar classes like "building" and "object."
Experimental Results
The model was evaluated on the ICCV 09 Dataset, consisting of 715 natural scene images across 8 classes.
Performance and Benchmarking
The Weighted Average ensemble method achieved the highest classification accuracy at 77.14%. This approach outperformed several established baselines:
- Baseline Pixel CRF: 74.3%
- Farabet et al. (CNN + Superpixel): 74.56%
- Tighe et al. (Features + Superpixel): 76.3%
The system excelled in identifying "Sky" (88.07%) and "Ground" (86.59%), though it struggled with the "Mountain" class (4% accuracy) due to a lack of representative training data in the dataset.
Real-Time Efficiency
By classifying superpixels rather than pixels, the system achieved a processing time of 25–30ms per image on a GTX 1080, making it suitable for real-time deployment.
Conclusion
This research demonstrates that combining multi-level neighborhood contexts significantly improves superpixel-level semantic labeling. The integration of Dempster-Shafer theory provides a robust mechanism for handling class confusion. Future work will focus on addressing data scarcity for rare classes and extending the framework to video segmentation to leverage temporal consistency.
Summary updated