3D Part Guided Image Editing for Fine-Grained Object Understanding

Zongdai Liu, Feixiang Lu, Peng Wang, Hui Miao, Liangjun Zhang, Ruigang Yang, Bin ZhouCVPR 2020 · OpenReview

Abstract

Holistically understanding an object with its 3D movable parts is essential for visual models of a robot to interact with the world. For example, only by understanding many possible part dynamics of other vehicles (e.g., door or trunk opening, taillight blinking for changing lane), a self-driving vehicle can be successful in dealing with emergency cases. However, existing visual models tackle rarely on these situations, but focus on bounding box detection. In this paper, we fill this important missing piece in autonomous driving by solving two critical issues. First, for dealing with data scarcity, we propose an effective training data generation process by fitting a 3D car model with dynamic parts to cars in real images. This allows us to directly edit the real images using the aligned 3D parts, yielding effective training data for learning robust deep neural networks (DNNs). Secondly, to benchmark the quality of 3D part understanding, we collected a large dataset in real driving scenario with cars in uncommon states (CUS), i.e. with door or trunk opened etc., which demonstrates that our trained network with edited images largely outperforms other baselines in terms of 2D detection and instance segmentation accuracy.

Introduction: The Challenge of Uncommon States

In the current landscape of autonomous driving (AD), standard vehicle detection is largely considered a solved problem. However, high-level safety requires more than just identifying a car's presence; it requires understanding its intent. If a parked car suddenly opens its door, a robot must anticipate a passenger stepping out. These "uncommon states" (CUS)—such as open trunks, bonnets, or specific light signals—are critical for safety but are significantly underrepresented in standard datasets like KITTI or CityScapes.

Example of a car in an uncommon state: flashing taillights indicating intent to stop.

This paper addresses two primary hurdles: the scarcity of real-world data for these rare events and the "domain gap" inherent in purely synthetic data. While 3D simulators can generate infinite variations, models trained on them often fail to generalize to the complex textures and lighting of the real world.

Methodology: 3D-Guided Image Editing

The authors propose a novel middle ground: 3D Part Guided Image Editing. Rather than synthesizing entire scenes, they "edit" real-world images to introduce uncommon states.

The Editing Pipeline

The process begins by fitting a 3D CAD model to a car instance in a real image using 6-DoF pose estimation. These 3D models are manually segmented into movable parts (doors, trunk, bonnet) and semantic parts (lights). By rotating these 3D parts and projecting them back into the 2D image plane, the authors can realistically "open" a door or "lift" a trunk. To maintain realism, they employ linear blending and bilateral filters to fill holes left by moving parts and use a photo-realistic rendering pipeline for newly visible regions, such as the reverse side of a door.

A variety of uncommon states (CUS) generated or identified across KITTI, CityScapes, and ApolloScape datasets.

Multi-Task Network Architecture

To leverage this data, a "Two-Backbone" architecture based on Mask-RCNN was developed. A main backbone is pre-trained on standard car datasets to capture general vehicle geometry, while an auxiliary backbone is pre-trained on COCO to better recognize the textures of the edited or rendered regions. The network simultaneously predicts 2D bounding boxes, instance masks, dynamic part masks, and a state description vector.

The CUS Dataset and Experimental Results

To benchmark this approach, the authors introduced the Cars in Uncommon States (CUS) dataset, featuring 1,441 real street-view images with 1,850 instances annotated across 12 defined states.

Experimental results demonstrate the efficacy of the 3D-guided editing approach:

  • Detection Accuracy: The proposed method achieved a 0.862 IoU for 2D detection on the CUS dataset, significantly outperforming a standard Mask-RCNN baseline (0.751 IoU).
  • Segmentation: Instance segmentation accuracy improved from 0.704 to 0.815 IoU.
  • Ablation Study: The dual-backbone strategy proved essential; freezing both backbones during training on edited data yielded a peak detection performance of 0.701 mAP, effectively bridging the gap between real and edited data.
  • Efficiency: The editing process is highly scalable, requiring only approximately 3 seconds per car instance.

Conclusion

By combining the realism of real-world imagery with the flexibility of 3D part manipulation, this work provides a scalable solution for training AD systems to handle rare but safety-critical scenarios. Future work will look to extend this framework to other road users, such as pedestrians, and incorporate temporal consistency for video-based understanding.

Summary updated

Abstract

Holistically understanding an object with its 3D movable parts is essential for visual models of a robot to interact with the world. For example, only by understanding many possible part dynamics of other vehicles (e.g., door or trunk opening, taillight blinking for changing lane), a self-driving vehicle can be success in dealing with emergency cases. However, existing visual models tackle rarely on these situations, but focus on bounding box detection. In this paper, we fill this important missing piece in autonomous driving by solving two critical issues. First, for dealing with data scarcity, we propose an effective training data generation process by fitting a 3D car model with dynamic parts to cars in real images. This allows us to directly edit the real images using the aligned 3D parts, yielding effective training data for learning robust deep neural networks (DNNs). Secondly, to benchmark the quality of 3D part understanding, we collected a large dataset in real driving scenario with cars in uncommon states (CUS), i.e. with door or trunk opened etc., which demonstrates that our trained network with edited images largely outperforms other baselines in terms of 2D detection and instance segmentation accuracy.
BibTeX
@inproceedings{liu2020d,
  author = {Zongdai Liu and Feixiang Lu and Peng Wang and Hui Miao and Liangjun Zhang and Ruigang Yang and Bin Zhou},
  title = {{3D Part Guided Image Editing for Fine-Grained Object Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=fqxJYVQ-hb},
}
Figure