TL;DR
This paper introduces a novel multi-scale Convolutional Neural Network (CNN) architecture for dense pixel-wise affordance segmentation in single RGB images. By effectively fusing mid-level visual cues like depth, surface normals, and semantic segmentation, their approach achieves state-of-the-art performance on a newly annotated dataset, demonstrating the critical role of these cues for understanding object affordances.
What Problem Does This Paper Address?
Understanding how objects and surfaces in a scene afford human actions (e.g., walking, sitting, reaching) is crucial for applications like robotics, autonomous driving, and human-computer interaction. This problem, known as affordance segmentation, is challenging because affordances are not inherent properties of objects but depend on their spatial context and relationships with the environment. Previous work often relied on holistic frameworks that iteratively refine affordance predictions and mid-level cues, or focused on image-level affordance prediction. This paper tackles the problem of dense pixel-wise affordance segmentation directly from a single RGB image, by proposing a feed-forward approach that explicitly infers and integrates multiple mid-level visual cues.
Methods
The core of the proposed method is a multi-scale CNN architecture designed to predict affordance maps. This architecture comprises four main components:
- Mid-level Cue Extraction: Three separate multi-scale CNNs are used to independently predict three crucial mid-level cues from the input RGB image:
- Depth Map: Estimating the distance of each pixel from the camera.
- Surface Normals: Determining the orientation of each surface.
- Coarse-level Semantic Segmentation: Classifying surfaces into broad categories like 'floor', 'structure', 'furniture', and 'props'.
- Feature Fusion and Affordance Prediction: A fourth multi-scale CNN takes the predicted mid-level cues and the original RGB image as input. This network fuses these features in a feed-forward manner to generate five distinct binary affordance maps, one for each of the target affordance types: 'walkable', 'sittable', 'lyable', 'reachable', and 'movable'.
The architecture leverages both "coarse-scale" networks (capturing global context with larger kernels) and "fine-scale" networks (capturing detailed features like object boundaries with smaller kernels) to effectively process information at different resolutions.
Datasets
The paper uses the NYUv2 dataset for evaluation. This dataset, originally containing RGBD images with pixel-wise depth maps and object class labels, was augmented by the authors with new, dense ground-truth annotations for the five affordance types. This augmentation was performed using a semi-automated method that leverages RGBD information and object labels, followed by manual correction and majority voting to ensure high-quality ground truth. The authors note that existing benchmark datasets were not suitable for their evaluation due to a lack of dense ground-truth affordance annotations.
Key Insights and Ideas
- Mid-level Cues are Crucial: The experiments strongly suggest that inferring mid-level cues (depth, surface normals, semantic labels) is essential for accurate affordance segmentation. Simply predicting affordances directly from RGB pixels yields poor results.
- Feed-Forward Integration: The paper demonstrates the effectiveness of a feed-forward approach for integrating mid-level cues, contrasting with prior iterative methods. This simplifies the pipeline and potentially improves efficiency.
- Context Matters for Affordances: The authors emphasize that affordances are not solely determined by object class but heavily depend on spatial context. For example, a chair might not be 'sittable' if it's placed under a table. Their method implicitly captures this context through the fused mid-level cues.
- Semi-Automated Ground Truth Generation: Developing a robust, semi-automated method for generating dense affordance annotations was a significant undertaking, addressing a key data bottleneck in the field.
Key Results
- Performance Improvement: The proposed multi-scale CNN architecture, which fuses depth, surface normals, and semantic labels, significantly outperforms baselines that omit one or more of these mid-level cues. For instance, the "w/o Depth" baseline shows a substantial drop in performance.
- Quantitative Evaluation: The paper reports pixel-wise Intersection over Union (IoU) accuracy on the NYUv2 dataset. Their approach achieves an average IoU of 49.56%, outperforming baselines that lack specific mid-level cues.
- Architecture Effectiveness: Using both coarse and fine-scale networks within the CNN architecture leads to better performance than using either scale alone, highlighting the benefit of multi-scale processing.
- Qualitative Results: Visualizations show that the model can predict affordance maps that align well with ground truth, demonstrating its ability to segment different affordance types across various indoor scenes.
Limitations and Future Work
- Failure Cases: The paper acknowledges failure cases, such as misclassifying a floor under a table as 'walkable'. This occurs when object clutter and partial occlusion lead to incorrect 3D geometry estimation, impacting affordance prediction.
- Data Augmentation: To overcome the limited size of the NYUv2 dataset, the authors employed data augmentation techniques. However, the performance might still be constrained by the dataset's inherent limitations.
- Future Work: While not explicitly detailed, the success of this approach suggests future work could explore incorporating more sophisticated scene understanding, handling dynamic scenes (videos), or extending to a wider range of affordance types and environments. The authors also mention that their method is designed to be efficient, with inference taking approximately 0.15 seconds per image on a K80 GPU.
Summary updated