Abstract
We present a scene parsing method that utilizes global context information based on both parametric and non-parametric models. Compared to previous methods that only exploit the local relationship between objects, we train a context network based on scene similarities to generate feature representations for global contexts. In addition, these learned features are utilized to generate global and spatial priors for explicit classes inference. We then design modules to embed the feature representations and the priors into the segmentation network as additional global context cues. We show that the proposed method can eliminate false positives that are not compatible with the global context representations. Experiments on both the MIT ADE20K and PASCAL Context datasets show that the proposed method performs favorably against existing methods.
The Challenge of Local Ambiguity
Scene parsing—the task of assigning a semantic label to every pixel in an image—is fundamentally difficult because local information is often ambiguous. A small patch of gray pixels, for instance, could represent a road, a sidewalk, a concrete wall, or beach sand depending on the broader environment. Standard Fully Convolutional Networks (FCNs) often struggle with these ambiguities because their receptive fields are too limited to capture the holistic scene category, leading to "impossible" predictions that violate global logic.
While previous attempts to solve this used post-processing like Conditional Random Fields (CRFs) or non-parametric image retrieval, these often relied on hand-crafted features. This research bridges the gap by embedding learned global context directly into the deep learning pipeline.
Methodology: Embedding Global Context
The proposed framework utilizes a two-pronged approach to capture and integrate global information into the segmentation process.
1. Global Context Network (Parametric)
The authors employ a Siamese VGG-16 network to learn a feature space where semantically similar scenes are clustered together. Rather than using rigid, human-defined scene categories, they define similarity based on pixel-label histograms. A key innovation here is Rare Class Enhancement: by re-weighting histograms so that rare objects (like a "microwave" or "tent") act as stronger anchors, the network learns a more discriminative similarity metric.
2. Prior Encoding (Non-Parametric)
Using the features from the Siamese network, the system retrieves the $K$ most similar images from the training set to generate two types of priors:
- Spatial Priors: Probability maps indicating where "stuff" classes (like sky or sea) typically appear geographically.
- Global Priors: Vectors indicating which "thing" classes (like person or car) are likely to exist anywhere in the scene.
3. Embedding Modules
These priors are not merely concatenated; they are processed through specific encoding modules. Global features act as a learnable bias term, while spatial priors are fused with final feature maps after passing through convolutional layers, allowing the network to "softly" weigh context against local evidence.
Key Insights
- Similarity over Classification: Learning a distance metric via a Siamese network captures the fluid nature of real-world scenes better than rigid classification labels.
- Rare Classes as High-Signal Cues: Common classes like "wall" provide little context, whereas rare classes are highly discriminative of the overall scene type.
- Priors as Soft Constraints: The network learns to ignore a prior if local evidence is overwhelming, but uses it to suppress logically inconsistent predictions (e.g., a "mountain" in a "bedroom").
Experimental Results
The method was evaluated on the MIT ADE20K and PASCAL Context benchmarks, showing consistent improvements over strong baselines. On ADE20K, adding global context to a DeepLab-ResNet101 baseline improved the Mean Intersection-over-Union (mIU) from 33.94% to 38.37%.
Qualitatively, the model excels at eliminating false positives. As seen in the beach scene above, the global context prevents the model from misidentifying sand as a wall or mountain—a common error for networks relying solely on local texture.
Conclusion
By combining parametric feature learning with non-parametric prior retrieval, this framework provides a robust mechanism for resolving local ambiguities in scene parsing. While the retrieval step adds a minor computational overhead (~0.7s per image), the significant gains in accuracy and logical consistency represent a major step forward for holistic scene understanding.
Summary updated