TL;DR
This paper introduces THEODORE, a large-scale synthetic dataset of 100,000 omnidirectional top-view indoor images with semantic segmentation and instance masks. They demonstrate that fine-tuning state-of-the-art CNNs on THEODORE significantly improves performance on real-world omnidirectional datasets, bridging the gap between synthetic and real-world data for this challenging domain.
What Problem Does This Paper Address?
Computer vision tasks like object detection and semantic segmentation have seen tremendous progress thanks to deep learning, particularly with Convolutional Neural Networks (CNNs). However, much of this progress has been driven by large datasets of perspective-view images. Omnidirectional cameras, which capture a much wider field of view, are becoming increasingly important for applications like autonomous navigation and surveillance. Yet, there's a significant lack of large-scale, diverse, and well-annotated datasets for training models on omnidirectional images, especially from a top-view perspective. Existing datasets often suffer from low in-class variance, missing ground truth, or limited scene variations. This scarcity hinders the development and deployment of robust computer vision systems for applications utilizing omnidirectional sensors.
Methods
The authors tackle this problem by creating a synthetic dataset called THEODORE. They leverage the Unity3D game engine to generate 3D virtual indoor environments (living rooms) populated with diverse objects and human characters.
Key methodological contributions include:
- Omnidirectional Image Generation: Since Unity3D's native camera is perspective, they employ a technique combining four perspective cameras with a 90° field of view each, then warping and combining these into a single fisheye-distorted image, mimicking real omnidirectional cameras.
- Synthetic Data Generation: Beyond RGB images, they generate pixel-accurate semantic segmentation and instance masks by replacing object shaders with unique colors based on class or object ID.
- Domain Randomization: To bridge the gap between synthetic and real data, they implement extensive domain randomization. This includes varying room textures, human character appearances (height, weight, clothing), object placements, and camera positions.
- Post-Processing: Bounding boxes are automatically extracted from the generated masks, and the data is converted into standard formats like TFRecords and PASCAL VOC2012.
- Transfer Learning Strategy: They fine-tune pre-trained CNNs (initially trained on MS COCO, a perspective dataset) using their synthetic THEODORE dataset. This transfer learning approach is central to their evaluation.
Datasets
- THEODORE (Training): A newly introduced synthetic dataset comprising 100,000 high-resolution, diversified fisheye images of indoor environments. It includes annotations for semantic segmentation, instance masks, and bounding boxes for 16 classes.
- MS COCO (Baseline Training): Used as a baseline for pre-training CNN models before fine-tuning on THEODORE.
- FES (Evaluation): A newly annotated real-world dataset of 301 fisheye top-view images with six class labels (person, armchair, chair, table, TV, wheeled walker). This serves as their primary evaluation benchmark for both object detection and semantic segmentation.
- HDA (Evaluation): A real-world dataset containing 1388 labeled images from a top-view camera, with barrel distortion, primarily used for person detection.
- Bomni (Evaluation): A real-world dataset from an omnidirectional camera in a single room, used for person tracking and action recognition, here adapted for person detection.
Key Insights and Ideas
- Synthetic Data as a Powerful Proxy: The paper strongly advocates for the effectiveness of synthetic data generation for training models on domains where real-world data is scarce. Their results show significant performance gains after fine-tuning on THEODORE, demonstrating that synthetic data can effectively compensate for the lack of real omnidirectional datasets.
- Domain Randomization is Crucial: The extensive use of domain randomization is highlighted as a key factor in achieving good generalization from synthetic to real data. By varying textures, object appearances, and camera poses, they create a more robust training signal.
- Top-View Omnidirectional is Underserved: The work explicitly points out the lack of datasets for this specific viewpoint and camera type, underscoring the importance of their contribution.
- Transfer Learning from Perspective to Omnidirectional: The success of fine-tuning models initially trained on perspective datasets (like MS COCO) to perform well on omnidirectional data is a critical finding, suggesting a viable path for leveraging existing resources.
Key Results
- Significant Performance Improvement: Fine-tuning on THEODORE led to substantial improvements in Average Precision (AP) for object detection across multiple meta-architectures (SSD, R-FCN, Faster R-CNN) on the FES dataset, outperforming MS COCO baselines. For the "person" class on FES, they achieved an AP up to 0.90.
- Semantic Segmentation Gains: For semantic segmentation on FES, fine-tuning with THEODORE resulted in a mean Intersection over Union (mIoU) of 0.36 for SegNet and 0.23 for PSPNet, showing promise for this task as well.
- Dataset Size vs. Variation: Experiments indicated that the number of scene variations (different textures and environments) had a greater impact on performance than the sheer number of images. A subset of 25k images with 500 scenes yielded the highest mAP.
Limitations and Future Work
- Class Imbalance and Variation: The authors acknowledge that further work is needed to balance the classes within THEODORE and increase variations in illumination, texture, and viewing angles for certain object classes (e.g., armchair, chair, TV) to improve their detection performance.
- Real-World Scene Diversity: While THEODORE is diverse, it's still synthetic. The authors plan to add more real-world indoor scenes to their evaluation suite.
- Additional Ground Truth: Future work will focus on generating more comprehensive ground truth data, including omnidirectional depth, skeletons, and optical flow, from rendered scenes.
- Texture Mismatch: For semantic segmentation, they hypothesize that the synthetic furniture textures might differ significantly from real-world textures, requiring further investigation.
Summary updated