Abstract
We propose an object clique potential for semantic segmentation. Our object clique potential addresses the misclassified object-part issues arising in solutions based on fully-convolutional networks. Our object clique set, compared to that yielded from segment-proposal-based approaches, is with a significantly smaller size, making our method consume notably less computation. Regarding system design and model formation, our object clique potential can be regarded as a functional complement to local-appearance-based CRF models and works in synergy with these effective approaches for further performance improvement. Extensive experiments verify our method.
Introduction
In the field of semantic segmentation, Fully Convolutional Networks (FCNs) have become a cornerstone due to their efficiency and end-to-end training capabilities. However, FCNs often suffer from a "part-whole" misclassification problem. Because their receptive fields are fixed and relatively local, they may misidentify parts of an object—for instance, labeling a dog's tail as a "cat" because its texture resembles feline fur in isolation.
Standard refinement techniques like Conditional Random Fields (CRFs) are effective at sharpening boundaries but often fail to correct these large-scale semantic errors. Since CRFs are "locally minded," they lack the global context necessary to flip a large blob of misclassified pixels back to the correct category.
The Object Clique Potential
To address these limitations, we introduce Object Clique Potentials, a global regularization term that bridges the gap between local pixel-wise predictions and object-level context. We formulate semantic segmentation as an energy minimization task with three distinct terms:
- Unary Potential ($\phi_1$): The standard pixel-wise class probability from an FCN.
- Local Appearance Potential ($\phi_3$): A standard CRF term that ensures neighboring pixels with similar colors have the same label (refining boundaries).
- Object Clique Potential ($\phi_2$): Our core innovation. This term encourages groups of pixels (cliques) to take labels consistent with an object-level detection score.
Efficient Clique Generation
Unlike previous segment-proposal methods that evaluate thousands of candidates, we generate a highly efficient set of cliques. By using the initial FCN output as a "seed," we merge adjacent regions into a hierarchical proposal tree. This results in an average of only 4 proposals per image, drastically reducing computational overhead while maintaining high quality.
Global Context and Synergy
The Object Clique Potential utilizes a co-occurrence prior to incorporate "common sense" into the model. For example, if the system detects both a "horse" and a "person," it gains confidence in both labels, whereas a "horse" and a "sofa" would be flagged as unlikely. This top-down view complements the bottom-up refinement of CRFs, allowing the model to fix both edges and large semantic "blobs" simultaneously.
Experimental Results
We evaluated our approach on the PASCAL VOC 2012 benchmark. Integrating the Object Clique Potential into a baseline FCN improved the Mean Intersection-over-Union (mIoU) from 62.48% to 64.07% on the validation set. When combined with state-of-the-art CRF models like DeepLab, our method achieved 71.2% mIoU on the test set, outperforming the baseline (70.9%).
Qualitatively, the method successfully resolves complex misclassifications. For instance, it can correctly separate a bird perched on a cow's back, whereas a standard FCN might merge them into a single object. While slower than a raw FCN, the method is roughly 10x faster than previous proposal-based methods like SDS.
Conclusion
Object Clique Potentials provide a computationally efficient way to inject global object context into semantic segmentation pipelines. By complementing local-appearance models, our method effectively addresses the "part-whole" problem, leading to both quantitative and qualitative improvements in segmentation accuracy. Future work will explore extending this framework to semi-supervised settings to reduce reliance on dense pixel-level annotations.
Summary updated