Scene-Aware Deep Networks for Semantic Segmentation of Images

Zhike Yi, Tao Chang, Shuai Li, Ruijun Liu, Jing Zhang, Aimin HaoIEEE Access 2019 · DOI

Abstract

Scene classification and semantic segmentation are two important research directions in computer vision. They are widely used in the research of automatic driving and human–computer interaction. The purpose of the scene classification is to use the image classification to determine the category of the scene in an image by analyzing the background and the target object, while semantic segmentation aims to classify the image at the pixel level and mark the position and semantic information of the scene unit. In this paper, we aimed to train the semantic segmentation neural network in different scenarios to obtain the models with the same number of scene categories, which they are used to process the images. During the process of the actual test, the semantic segmentation dataset was firstly divided into three categories based on the scene classification algorithm. Then the semantic segmentation neural network is trained under three scenarios, and three semantic segmentation network models are obtained accordingly. To test the property of our methods, the semantic segmentation models we got were selected to treat other pictures, and the results obtained from the performance of scene-aware semantic segmentation were much better than semantic segmentation without considering categories. Our study provided an essential improvement of semantic segmentation by adding category information into consideration, which will be helpful to obtain more precise models for further picture analysis.

Introduction

Semantic segmentation—the task of assigning a class label to every pixel in an image—is a cornerstone of computer vision, essential for applications like autonomous driving and human-computer interaction. Despite significant progress with architectures like Fully Convolutional Networks (FCNs) and DeepLab, models often suffer from "relationship mismatch" errors. These occur when a model lacks the global context to distinguish between similar-looking objects in different environments, such as confusing a bed in an indoor scene with a feature in an outdoor park.

The authors argue that integrating global scene context as a "prior" can significantly reduce these errors. By knowing the type of scene beforehand, a model can narrow its search space and improve the precision of object boundaries.

Methodology

The researchers propose a Scene-Aware Deep Network pipeline that employs a "divide and conquer" strategy. The process is divided into three distinct phases: scene classification, specialized training, and a scene-aware test phase.

Specialized Training Phase

To avoid a "one-size-fits-all" approach, the authors trained multiple specialized models. The process begins with a "pre-training" step on the entire dataset to establish a baseline of features. Subsequently, the dataset is divided into three broad categories: Indoor, Outdoor Natural, and Outdoor Artificial. The models are then fine-tuned specifically on these subsets.

Specialized training phase: models are pre-trained on the full dataset and then fine-tuned on specific scene categories to capture specialized context.

A critical component of this phase is proportional training time allocation. To handle dataset imbalances, training iterations were allocated based on the ratio of images in each category (approximately 5:1:4), preventing overfitting on smaller subsets while ensuring larger ones were sufficiently learned.

Scene-Aware Inference

During the test phase, a new image is first processed by a VGGNet-based scene classifier. The resulting category label (e.g., "Indoor") acts as a switch, directing the image to the corresponding specialized segmentation model (e.g., a DeepLab model trained specifically on indoor scenes).

The inference workflow: an input image is first classified into a scene category, which determines the specialized segmentation model used for the final output.

Experimental Setup

The study utilized the ADE20K dataset, a rigorous benchmark for scene parsing containing 150 object categories. The training set comprised approximately 20,210 images, with 1,611 images reserved for testing. The authors manually and algorithmically partitioned this dataset into the three scene categories to facilitate their specialized training regime.

Key Insights

  • Context as a Constraint: Scene information effectively filters out impossible class assignments, reducing classification noise.
  • Semantic Importance: A control experiment using random category assignments resulted in worse performance than the baseline, proving that the semantic meaning of the scene (not just the split itself) is the driver of improvement.
  • Efficiency through Balancing: Proportional iteration allocation was found to be superior to equal time allocation across models.

Results

The scene-aware approach demonstrated significant quantitative improvements across different base architectures:

  • Classification Accuracy: The VGGNet classifier achieved 96.9% accuracy, ensuring high reliability in model selection.
  • DeepLab v2: mIoU increased from 28.93% (baseline) to 33.68%.
  • DeepLab v3+: Pixel accuracy rose to 79.45% (from 71.95%) and mIoU reached 37.86% (from 36.23%).
  • Theoretical Potential: Using ground truth labels for scene classification pushed the mIoU to 38.27%, indicating that further improvements in classification could yield even better segmentation results.

Conclusion

While the scene-aware method introduces overhead in terms of training time and model storage, the gains in accuracy and boundary precision are substantial. Future work may explore more granular scene categories or the injection of other types of a priori information to further refine pixel-level analysis.

Summary updated

Abstract

Scene classification and semantic segmentation are two important research directions in computer vision. They are widely used in the research of automatic driving and human–computer interaction. The purpose of the scene classification is to use the image classification to determine the category of the scene in an image by analyzing the background and the target object, while semantic segmentation aims to classify the image at the pixel level and mark the position and semantic information of the scene unit. In this paper, we aimed to train the semantic segmentation neural network in different scenarios to obtain the models with the same number of scene categories, which they are used to process the images. During the process of the actual test, the semantic segmentation dataset was firstly divided into three categories based on the scene classification algorithm. Then the semantic segmentation neural network is trained under three scenarios, and three semantic segmentation network models are obtained accordingly. To test the property of our methods, the semantic segmentation models we got were selected to treat other pictures, and the results obtained from the performance of scene-aware semantic segmentation were much better than semantic segmentation without considering categories. Our study provided an essential improvement of semantic segmentation by adding category information into consideration, which will be helpful to obtain more precise models for further picture analysis.
BibTeX
@inproceedings{yi2019scene,
  author = {Zhike Yi and Tao Chang and Shuai Li and Ruijun Liu and Jing Zhang and Aimin Hao},
  title = {{Scene-Aware Deep Networks for Semantic Segmentation of Images}},
  booktitle = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2918700},
}
Figure