A Dataset for Developing and Benchmarking Active Vision

Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Kosecka, Alexander C. Berg

Abstract

We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning.

Introduction

The rapid progress in computer vision has largely been driven by static image datasets like ImageNet and COCO. However, robotic agents do not merely observe static snapshots; they interact with and move through 3D environments. Standard datasets often suffer from "photographer bias," where objects are centered and well-lit. In contrast, a robot in a real-world setting might encounter small, occluded objects from unconventional angles.

To improve performance in these scenarios, a robot must employ "active vision"—moving to obtain a more informative viewpoint. Historically, testing active vision algorithms required expensive physical hardware or simplified synthetic simulations. The Active Vision Dataset (AVD) bridges this gap by providing a dense, real-world dataset that enables virtual navigation and benchmarking of active vision systems.

The Active Vision Dataset (AVD)

The primary contribution of this work is the AVD, which contains over 20,000 RGB-D images and 54,247 bounding boxes across 33 common object instances. The data was collected across nine unique indoor environments, including kitchens, offices, and living rooms.

Dense Sampling for Virtual Navigation

To allow a virtual robot to move realistically, the researchers captured images on a 30cm grid, rotating the camera 30 degrees at each point. This density ensures that the change in visual information between adjacent points is small enough for a robot to learn meaningful movement policies.

Comparison of camera sampling density between AVD (Ours) and existing datasets like NYUD2 and RGBD Scenes v2.

Efficient 3D-to-2D Labeling

Labeling tens of thousands of bounding boxes manually is prohibitively expensive. Instead, the authors created 3D reconstructions of the scenes using structure-from-motion (COLMAP) and dense reconstruction (PMVS). Objects were labeled once in 3D space, and these labels were then projected back into every 2D camera frame, ensuring perfect consistency across viewpoints.

A 3D reconstruction of an office scene used for automated 2D label generation.

Active Vision System and Results

The authors developed a baseline active vision system using a REINFORCE policy gradient algorithm. The system predicts the "next best move" (e.g., move forward, rotate) to improve its classification confidence.

Key Insights

  • Viewpoint Sensitivity: State-of-the-art detectors like SSD are highly sensitive to viewpoint. A detector might have 90% confidence from the front but fail entirely when viewing an object from the side.
  • Performance Killers: Quantitative analysis identified object scale and occlusion as the primary factors degrading detector performance in robotic settings.

Experimental Results

The reinforcement learning agent significantly outperformed "Random" and "Move Forward" baselines. In one test split, classification accuracy improved from 0.30 (single view) to 0.51 after 20 moves. In another, it rose from 0.42 to 0.73. Qualitatively, the agent learned to move closer to small objects or rotate to find unoccluded faces.

Conclusion and Future Work

The AVD provides a robust benchmark for active vision, demonstrating that virtual navigation in real-world imagery is a viable path for training robotic agents. Future work will focus on information fusion—using recurrent neural networks (RNNs) to aggregate data across an entire path rather than just the final frame. Additionally, expanding the task from recognition to full-room object search and increasing the diversity of scenes will be critical next steps.

The dataset and toolbox are available at: cs.unc.edu/~ammirato/active_vision_dataset_website/

Summary updated

Abstract

We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning. Our dataset is available for download at cs.unc.edu/~ammirato/active_vision_dataset_website/.
BibTeX
@inproceedings{ammirato2017dataset,
  author = {Phil Ammirato and Patrick Poirson and Eunbyung Park and Jana Kosecka and Alexander C. Berg},
  title = {{A Dataset for Developing and Benchmarking Active Vision}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2017},
  url = {https://openreview.net/forum?id=esecDohASvI},
}
Figure