Scene Parsing with Global Context Embedding

Wei-Chih Hung, Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang

Introduction

Scene parsing, the task of assigning a semantic category to every pixel in an image, remains a significant challenge in computer vision. While standard Fully Convolutional Networks (FCNs) excel at identifying local patterns, they often struggle with ambiguity when local information is insufficient. For instance, a patch of gray pixels could represent a road, a wall, or a sidewalk depending on the broader environment. This paper introduces a novel approach to bridge this gap by embedding global context directly into the segmentation process, effectively teaching neural networks to understand the "holistic" view of a scene.

Methodology

The proposed framework utilizes a two-pronged strategy to capture and integrate global context:

  1. Parametric Global Context Network: A Siamese network is trained to learn a continuous semantic similarity space. Rather than forcing images into rigid categories, the network learns to measure the distance between images based on their pixel-label distributions using a Ground Truth Distance metric (Chi-square distance of label histograms).
  2. Non-parametric Prior Generation: For any given input, the model retrieves the $K$-nearest similar images from the training set using the learned embeddings. These neighbors are used to generate two types of priors:
  3. Spatial Priors: Maps indicating the typical location of "stuff" (uncountable regions like sky or ground).
  4. Global Priors: Probability distributions of "things" (countable objects like people or cars) likely to be present anywhere in the scene.
  5. Feature and Prior Encoding: These global cues are integrated into standard segmentation architectures (e.g., FCN-8s, DeepLab) via specialized modules, allowing the model to refine local predictions based on the overall scene "vibe."

A Rare Class Enhancement mechanism is also included to ensure that small but contextually vital objects, such as a microwave in a kitchen, are not overshadowed by dominant classes like walls or floors.

Results and Evaluation

The researchers evaluated their method on two major benchmarks: MIT ADE20K (150 classes) and PASCAL Context (60 classes). The integration of global context led to consistent performance gains:

  • MIT ADE20K: The best-performing model (DeepLab-ResNet101 with Feature and Prior modules) achieved a 38.37% Mean IU, a significant improvement over the 33.94% baseline.
  • PASCAL Context: The method reached 46.52% Mean IU, outperforming several contemporary state-of-the-art models.
Qualitative comparison showing how global context eliminates local misclassifications (e.g., mistaking sand for mountain/building).

Qualitatively, the most striking improvement is the removal of "false positive" blobs—random patches of incorrect labels that are contextually inconsistent with the rest of the image. As shown in the figure above, the global context acts as a "sanity check," preventing the model from misidentifying a beach scene's sand as a building wall.

Insights and Future Work

The core insight of this work is that similarity is often more robust than categorization. By learning a continuous similarity space rather than discrete scene labels, the model becomes more flexible. Furthermore, treating "stuff" and "things" differently allows for more effective use of spatial versus global priors.

While effective, the non-parametric retrieval step introduces computational overhead (approximately 1.4 seconds per image) and relies on the diversity of the training set. Future research may focus on deeper integration of these global features during the initial training of base networks to further reduce latency and improve accuracy.

Summary updated

Abstract

We present a scene parsing method that utilizes global context information based on both the parametric and non- parametric models. Compared to previous methods that only exploit the local relationship between objects, we train a context network based on scene similarities to generate feature representations for global contexts. In addition, these learned features are utilized to generate global and spatial priors for explicit classes inference. We then design modules to embed the feature representations and the priors into the segmentation network as additional global context cues. We show that the proposed method can eliminate false positives that are not compatible with the global context representations. Experiments on both the MIT ADE20K and PASCAL Context datasets show that the proposed method performs favorably against existing methods.
BibTeX
@inproceedings{hung2017scene,
  author = {Wei-Chih Hung and Yi-Hsuan Tsai and Xiaohui Shen and Zhe Lin and Kalyan Sunkavalli and Xin Lu and Ming-Hsuan Yang},
  title = {{Scene Parsing with Global Context Embedding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.287},
}
Figure