Scene Parsing via Integrated Classification Model and Variance-Based Regularization

Hengcan Shi, Hongliang Li, Qingbo Wu, Zichen SongCVPR 2019 · DOI

Researchers at the University of Electronic Science and Technology of China have introduced a novel approach to address the "identity crisis" pixels often face in scene parsing. By transitioning from a single-step classification approach to a multi-stage process and introducing an economic-inspired regularization technique, the authors achieved state-of-the-art results on major benchmarks.

The Problem: Across-Category Competition

Scene parsing involves assigning a semantic label to every pixel in an image. Standard models often struggle when categories are visually similar, such as "ground" (outdoors) and "floor" (indoors). Most existing models utilize a single multinomial classifier, which creates "across-category competition." If a model is slightly more confident in an incorrect but similar category, the score for the correct category is suppressed so heavily that recovery becomes nearly impossible. This leads to severe misclassifications where the correct answer is buried under a competing label.

Figure 1: Comparison between a general-classifier-based model (a) and the proposed integrated classification model (b). The integrated model correctly distinguishes between 'ground' and 'floor' by refining initial predictions.

The Integrated Classification Model (ICM)

The proposed Integrated Classification Model (ICM) breaks the classification process into three distinct stages to preserve information and refine predictions:

  1. Feature Extraction: A standard backbone (e.g., ResNet or PSPNet) encodes the input image into high-dimensional feature maps.
  2. General Classification: Instead of a single multinomial classifier, the model employs multiple binary classifiers. Each classifier independently determines if a pixel belongs to its specific category. This prevents high scores in one category from automatically suppressing others, keeping the correct class "alive" in the probability distribution even if it isn't the top choice initially.
  3. Refinement Classification: This network processes the original features alongside the scores from the general classification stage. It acts as a specialist, utilizing spatial context (via $3 \times 3$ convolutions) to differentiate between categories that received high initial scores.
Figure 6: The detailed architecture of the proposed system, consisting of (1) Feature Extraction, (2) General Classification, and (3) Refinement Classification networks.

Variance-Based Regularization (VBR)

To enhance model robustness, the researchers introduced Variance-Based Regularization (VBR). Inspired by the Herfindahl-Hirschman Index (HHI)—a measure used in economics to determine market concentration—VBR encourages the model to maximize the variance between class scores. Essentially, it tells the model: "Don't just pick a winner; make sure the winner is significantly ahead of the runners-up." This forces the model to be more "decisive," thereby reducing ambiguity between confusing classes.

Experimental Results

The ICM was evaluated on three widely used scene parsing benchmarks: NYU Depth v2, PASCAL-Context, and SUN-RGBD. The proposed method outperformed several heavyweights in the field, including PSPNet and DeepLab v3+.

  • NYU Depth v2: The model achieved 75.4% pixel accuracy (pAcc) and 50.7% mean Intersection-over-Union (mIoU), a significant jump over the baseline PSPNet (73.6% pAcc / 46.9% mIoU).
  • PASCAL-Context: Reached 80.5% pAcc, outperforming DeepLab v3+ by 5%.
  • Ablation Studies: Results indicated that the ICM provided the primary performance boost (~1.4% pAcc), while the VBR loss added a consistent improvement of 0.4% to 0.8% across the board.

Conclusion and Future Work

The Integrated Classification Model and Variance-Based Regularization effectively mitigate misclassifications between confusing categories by moving away from the "zero-sum game" of multinomial softmax. While the model shows significant improvements, challenges remain regarding over-segmentation of multi-colored objects and the handling of delicate object boundaries. Future research may focus on integrating multi-scale methods to better capture complex structures and fine details simultaneously.

Summary updated

BibTeX
@inproceedings{shi2019scene,
  author = {Hengcan Shi and Hongliang Li and Qingbo Wu and Zichen Song},
  title = {{Scene Parsing via Integrated Classification Model and Variance-Based Regularization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00545},
}
Figure