Abstract
Semantic segmentation models often struggle with class imbalance, frequently prioritizing majority classes like "road" over critical minority classes such as "pedestrians." This bias leads to dangerous false negatives in safety-critical applications like autonomous driving. We propose MetaFusion, a post-processing framework that utilizes a Maximum Likelihood (ML) decision rule to increase sensitivity to rare classes, followed by an uncertainty-based meta-classifier to filter the resulting false positives. Our approach requires no retraining and achieves a significantly better trade-off between false negatives and false positives than standard interpolation methods on the Cityscapes dataset.
Introduction
In semantic segmentation, the goal is to assign a class label to every pixel. However, standard training objectives minimize overall error, which favors frequent classes. For an autonomous vehicle, missing a pedestrian (a False Negative) is far more hazardous than misidentifying a sidewalk (a False Positive). While increasing sensitivity via thresholding is possible, it typically results in an unmanageable surge of false alarms. MetaFusion addresses this by identifying "disputed" segments where standard Bayesian predictions and ML-based predictions disagree, then resolving these disputes using a secondary classifier.
Methodology
The MetaFusion pipeline consists of three primary stages:
- Maximum Likelihood (ML) Decision Rule: Standard models employ the Bayes (MAP) rule, which is biased by class priors. By dividing the softmax output by class priors, we derive an ML rule that is inherently more sensitive to rare classes. This step "uncovers" objects previously missed but introduces "ghost" detections.
- Meta-Classification: We employ a Gradient Boosting Tree model to evaluate segments where the ML rule and Bayes rule disagree. This meta-model acts as a filter, deciding whether to "believe" the more sensitive ML prediction or stick with the conservative Bayes prediction.
- Uncertainty and Context Fusion: The meta-classifier utilizes a variety of features, including pixel-wise entropy (dispersion), segment geometry, and spatial context (neighboring class statistics).
Experimental Results
We evaluated MetaFusion on the Cityscapes dataset, focusing on the "human" class. Using a MobileNetV2 backbone, MetaFusion successfully reduced the number of false positives to approximately one-third of those generated by a pure ML rule, while retaining nearly 80% of the additional true detections.
As shown in the performance analysis, MetaFusion consistently outperforms standard Bayes-ML interpolation, providing a more efficient Pareto front for the trade-off between false negatives and false positives.
Key Insights
- Agreement as a Signal: High agreement between Bayes and ML rules indicates high confidence. Disagreement zones are the primary areas where rare objects are likely to be found.
- Contextual Importance: Spatial context—such as the proximity of "bicycle" pixels to a "human" segment—proved to be a highly predictive feature for the meta-classifier, helping distinguish real objects from noise.
- Architectural Agnostic: Because MetaFusion is a post-processing step, it can be integrated with any existing segmentation network (e.g., DeepLabV3+, Xception) without retraining.
Conclusion
MetaFusion provides a robust, computationally efficient method for balancing class sensitivity in semantic segmentation. By leveraging meta-classification on disputed segments, it offers a controlled way to reduce critical false negatives. Future work will explore the integration of temporal dynamics from video data to further refine the meta-classifier's accuracy.
Summary updated