Abstract
Semantic segmentation in cluttered indoor environments remains a challenging task due to the high variability of object appearances. While depth sensors provide valuable 3D information, standard probabilistic models often fail to enforce basic physical consistency, leading to "impossible" predictions. This paper introduces a novel framework that augments Conditional Random Fields (CRFs) with hard mutual exclusion (mutex) constraints. By encoding geometric and logical rules—such as vertical relationships and object co-occurrence—into an Integer Quadratic Programming (QP) optimization, the proposed method significantly improves segmentation accuracy and generalization across diverse datasets.
Introduction
The goal of semantic segmentation is to assign a categorical label to every pixel in an image. In indoor scenes, this task is complicated by clutter, occlusion, and lighting variations. The introduction of RGB-D sensors like the Microsoft Kinect provided a new dimension of data, yet early approaches often treated depth as a simple additional feature channel. This frequently resulted in physically nonsensical outputs, such as a "floor" label appearing above a "table."
The authors argue that the rigid geometric laws of the physical world should be treated as hard constraints rather than soft suggestions. By pruning the search space of possible labelings to only those that are physically plausible, the model can achieve higher accuracy and better robustness.
Methodology: Quadratic Mutex Constraints
The proposed framework treats scene labeling as an optimization problem over image segments (superpixels). The core innovation is the use of Quadratic Mutex Constraints (QMC) within an Integer Quadratic Programming (QP) formulation. These constraints are represented by a binary matrix $M$, where $M_{ij} = 1$ indicates that a specific pair of labels for two regions is physically or logically impossible.
The paper defines three primary classes of constraints:
- Global Object Co-occurrence: Prevents the simultaneous labeling of objects that rarely appear together (e.g., a toilet and a stove).
- Relative Height Relationships: Enforces vertical logic based on 3D data (e.g., a ceiling must be higher than a wall).
- Local Support Relationships: Models physical stability, ensuring that supported objects (like a monitor) are correctly positioned relative to their supporters (like a desk).
To handle the computational complexity, the authors relax the integer QP into a continuous domain and employ a line search algorithm to recover a valid binary solution. To maintain efficiency, the number of candidate labels per superpixel is limited to $k=5$.
Key Insights: Absolute Height and Gravity Alignment
A critical observation in this work is the distinction between relative and absolute height. Relative height—measured from the lowest point in a single frame—can be highly misleading if the floor is not visible. To address this, the authors estimate the absolute camera height to establish a consistent, gravity-aligned coordinate system across the entire dataset.
By grounding the geometric constraints in an absolute coordinate system, the model becomes significantly more robust to camera tilt and varying viewpoints. This "common sense" approach allows the model to generalize effectively even when visual features vary between datasets.
Experimental Results
The model was evaluated on two major benchmarks:
- NYU-Depth V2: On this 40-class segmentation task, the Mutex approach achieved 63.8% pixel accuracy, outperforming the previous state-of-the-art by 3.5%.
- SUN3D Generalization: To test the universality of the learned constraints, the authors applied the NYU-trained model to SUN3D keyframes without further training. It achieved 65.7% pixel accuracy, demonstrating that physical laws (e.g., "floors are at the bottom") are consistent across different environments.
Ablation studies confirmed that co-occurrence and relative height constraints were the most significant contributors to the performance gains.
Limitations and Conclusion
While the method sets a strong precedent for "physics-aware" computer vision, it relies on the quality of initial superpixel segmentation (gPb-UCM). If the initial segments are flawed, the mutex constraints cannot correct the underlying error. Additionally, solving the QP remains computationally intensive.
In conclusion, by integrating hard physical constraints into the segmentation process, this work demonstrates that geometry is more than just a feature—it is a rulebook for scene understanding. Future research may focus on integrating these constraints directly into deep learning architectures to further enhance real-time performance and accuracy.
Summary updated