TL;DR
This paper introduces "Task Driven Object Detection," a new problem formulation that goes beyond simply identifying object categories to selecting the most suitable object for a given task. They introduce the COCO-Tasks dataset and propose a Gated Graph Neural Network (GGNN) approach that leverages both object appearance and scene context to solve this problem, outperforming existing methods.
What Problem Does This Paper Address?
Traditional object detection, while impressive, focuses on identifying instances of predefined categories (e.g., "chair," "bottle"). However, for robots or autonomous systems interacting with the real world, simply detecting all chairs isn't enough. If a robot needs to serve wine, it must not only detect wine glasses but also choose the best one, considering factors like whether it's clean, already in use, or if a cup is a more suitable alternative if no wine glasses are present. This nuanced decision-making, which humans do effortlessly, is the core of "task-driven object detection" and is not addressed by current benchmarks.
Methods
The authors propose a novel approach that combines object detection with scene context modeling.
- Object Feature Extraction: They use a pre-trained ResNet101 to extract visual features from each detected object's bounding box.
- Gated Graph Neural Network (GGNN): A GGNN is employed to model the relationships between all detected objects in an image. Each object becomes a node in the graph, and the GGNN iteratively updates each node's hidden state by aggregating information from all other nodes. This allows the model to understand the global scene context.
- Task-Specific Prediction: The GGNN's final hidden states, along with initial object features and class information, are used to predict the probability of each object being suitable for a given task.
- Fusion: They combine predictions from the GGNN and a direct visual feature-based classifier to improve robustness, especially in the presence of noisy detections.
Datasets
- COCO-Tasks Dataset: A new dataset introduced by the authors, built upon the existing COCO dataset. It contains approximately 40,000 images annotated for 14 diverse tasks (e.g., "serve wine," "sit comfortably," "dig hole"). For each task, human annotators marked the most suitable objects. This dataset differs from standard object detection datasets by focusing on task-specific suitability rather than just category presence.
- COCO Dataset: Used as the base for the COCO-Tasks dataset and for training/evaluating baselines.
Key Insights and Ideas
- Task-driven object selection is fundamentally different from standard object detection: Simply training a detector for each task's preferred object category performs poorly because it ignores the crucial aspect of relative suitability and scene context.
- Scene context is paramount: The decision of which object is "best" often depends on what other objects are present in the scene and their properties. For example, a chair might be preferred over a table for "stepping on something" if the chair is more accessible.
- Instance-level attributes matter: Sometimes, the choice isn't just about the object category but about the specific attributes of an instance (e.g., the height of a bed).
- GGNNs are effective for modeling inter-object relationships: The proposed GGNN successfully captures scene context, leading to significant performance improvements.
- Fusion of different prediction sources (visual features and GGNN context) enhances robustness: Combining these sources helps overcome limitations of individual methods, particularly when dealing with imperfect object detections.
Key Results
- The proposed GGNN-based method achieves a mean Average Precision ([email protected]) of 32.6% on Faster-RCNN detections, significantly outperforming baselines.
- On ground truth bounding boxes, the method achieves 74.2% [email protected], demonstrating its strong potential with perfect detections.
- The strongest baseline, a task-specific classifier, achieved 28.8% [email protected] on detections, highlighting the advantage of the proposed context-aware approach.
- Ablation studies showed that joint training, direct class input, and the GGNN for scene context modeling all contribute to performance gains. Weighted aggregation further improved results on detections.
Limitations and Future Work
- Noisy Detections: While the proposed method handles noisy detections better than baselines, performance still drops significantly compared to using ground truth bounding boxes. The authors suggest that their weighted aggregation mechanism helps mitigate this.
- Visual Input Importance: Experiments removing visual input showed a significant drop in performance, emphasizing that object appearance is crucial.
- Future Work: The paper suggests exploring more sophisticated ways to model object relationships and potentially incorporating temporal information for video-based tasks. Further investigation into how to best leverage instance-level attributes for finer-grained selection is also a promising direction.
Summary updated