A dataset for developing and benchmarking active vision

Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Kosecka, Alexander C. Berg

TL;DR

The authors introduce the Active Vision Dataset (AVD), a large-scale collection of dense RGB-D imagery designed to simulate robotic movement in real-world indoor environments. By providing a "virtual" environment where researchers can test "next best move" algorithms without a physical robot, the paper establishes a new benchmark for active object recognition using reinforcement learning.

What Problem Does This Paper Address?

Most computer vision progress has happened in the context of static image datasets (like ImageNet or COCO). However, robots don't just look at static snapshots; they move through 3D space.

Standard datasets suffer from "photographer bias"—objects are usually centered, well-lit, and viewed from the front. In contrast, a robot might see a tiny, occluded sliver of a "Red Bull" can from a weird angle. To improve, the robot needs to move. The challenge is that testing "active vision" (moving to get a better look) usually requires expensive physical robots or simplified synthetic simulations. This paper bridges that gap by providing a dense, real-world dataset that allows for "virtual" navigation.

Methods

The researchers collected over 20,000 RGB-D images across nine unique indoor scenes (kitchens, offices, living rooms).

  • Dense Sampling: Instead of random snapshots, they captured images on a 30cm grid, rotating the camera 30 degrees at each point. This density allows a "virtual robot" to move through the scene by simply switching between adjacent images.
  • 3D-to-2D Labeling: Rather than labeling 50,000+ bounding boxes by hand, they created 3D reconstructions of the scenes. They labeled objects once in 3D space and then projected those labels back into every 2D camera frame, ensuring perfect consistency.
  • Active Vision System: They built a baseline system using a REINFORCE policy gradient algorithm. The system takes a current view, predicts the "next best move" (forward, rotate, etc.) to improve its confidence, and receives a reward if it correctly classifies the object at the end of its path.

Datasets

  • Active Vision Dataset (AVD): The primary contribution. It features 20,000+ RGB-D images and 54,247 bounding boxes of 33 common object instances.
  • BigBIRD: Used to train the instance classifier, as it provides high-quality masks of objects that the authors overlaid onto random backgrounds for data augmentation.
  • Comparison Datasets: The paper notes that previous datasets like NYUD2 or RGBD Scenes v2 are too sparsely sampled to allow for simulated robotic navigation.

Key Insights and Ideas

  • The "Viewpoint Gap": The authors demonstrate that state-of-the-art detectors (like SSD) are incredibly sensitive to viewpoint. A detector might have 90% confidence from the front but 0% from the side. This "instability" is exactly why active vision is necessary.
  • Virtual Navigation is Viable: They proved that a 30cm/30-degree sampling rate is "dense enough." The change in detection scores between these points is small enough that a robot can learn meaningful movement policies.
  • Efficiency of 3D Labeling: By using structure-from-motion (COLMAP) and dense reconstruction (PMVS), the authors show that 3D labeling is the only scalable way to create ground truth for video-like robotic datasets.

Key Results

  • Detection Sensitivity: Quantitative tests showed that object scale and occlusion are the primary "performance killers" for standard detectors in robotic settings.
  • Active vs. Passive: The reinforcement learning agent significantly outperformed "Random" and "Move Forward" baselines.
  • Accuracy Gains: In one test split, the system's classification accuracy started at 0.30 (single view) and climbed to 0.51 after 20 moves. In another split, it jumped from 0.42 to 0.73.
  • Qualitative Success: The agent learned intuitive behaviors, such as moving closer to small objects or rotating around an object to find a non-occluded face.

Limitations and Future Work

  • Information Fusion: Currently, the system only uses the final image to make a classification. The authors suggest that future work should use recurrent neural networks (RNNs) to aggregate information from the entire path of images.
  • Search vs. Recognition: The current benchmark provides the agent with the initial bounding box. A more difficult, realistic task would involve the robot searching for the object in a room from scratch.
  • Dataset Scale: While 20,000 images is a great start, 9 scenes is still a small sample of the infinite variety of human indoor environments.

For more details, the dataset and toolbox are available at: cs.unc.edu/~ammirato/active_vision_dataset_website/

Summary updated

Abstract

We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning. Our dataset is available for download at cs.unc.edu/~ammirato/active_vision_dataset_website/.
BibTeX
@inproceedings{ammirato2017dataset,
  author = {Phil Ammirato and Patrick Poirson and Eunbyung Park and Jana Kosecka and Alexander C. Berg},
  title = {{A dataset for developing and benchmarking active vision}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2017},
  url = {https://openreview.net/forum?id=esecDohASvI},
}

Loading related content...

Figure