Amodal Completion and Size Constancy in Natural Scenes

Abhishek Kar, Shubham Tulsiani, João Carreira, Jitendra Malik

Abstract

Standard object detection systems often fail to account for the physical reality of a 3D world, interpreting occluded objects as smaller versions of themselves. This paper presents a novel framework that moves beyond 2D bounding boxes to estimate the true physical size and relative depth of objects from a single image. By combining amodal completion—predicting the full extent of partially hidden objects—with geometric scene priors and deep-learning-based focal length estimation, the authors demonstrate how computers can achieve "size constancy," the human ability to perceive an object's true size regardless of its distance or partial occlusion.

Introduction: The Challenge of 2D Perspective

Most computer vision systems are "modal," meaning they only process visible pixels. If a person stands behind a counter, a standard detector draws a box around the visible torso, effectively seeing a "short" human. Furthermore, perspective projection creates scale ambiguity: a small object near the lens can appear identical in size to a large object far away. To achieve a 3D-like understanding from 2D images, systems must overcome three hurdles: occlusion, scale ambiguity, and the lack of camera calibration (focal length) in "in-the-wild" photographs.

Methodology

The researchers proposed a three-stage pipeline to bridge the gap between 2D pixels and 3D physical dimensions.

Amodal Bounding Box Completion

The first stage involves a Convolutional Neural Network (CNN) trained to predict the full bounding box of an object, including invisible parts. For instance, if only the top half of a person is visible, the model regresses the coordinates for where the feet should be. Interestingly, a class-agnostic model performs well here, suggesting that occlusion patterns are universal across different object types.

Amodal completion predicts the full extent of occluded objects, such as the person's legs behind the counter.

Geometric Reasoning and Familiar Size

By assuming objects rest on a common ground plane, the authors formulated an optimization problem using a simplified perspective camera model. They leveraged "familiar size" priors—the knowledge that humans or chairs have predictable physical heights—to solve for camera height, the horizon line, and the physical heights of all detected objects simultaneously.

Focal Length Prediction via Scene Recognition

To resolve final scale ambiguities, the authors hypothesized that scene type (e.g., "bedroom" vs. "forest") correlates with typical photography styles and focal lengths. They trained a CNN on the Places dataset to categorize images into focal length "bins," providing a proxy for camera intrinsics.

Results and Discussion

The system was evaluated using datasets like PASCAL 3D+ and PASCAL VOC 2012. The amodal completion model significantly outperformed standard modal boxes on occluded objects, improving Mean IoU from 0.59 to 0.62. The focal length predictor also proved highly effective, achieving a top-5 misclassification rate of just 3.1%, confirming that scene category is a powerful proxy for camera intrinsics.

More importantly, the system successfully estimated veridical sizes in meters. As shown in the visualization below, the system correctly identifies that a person is taller than a chair, even when the chair is closer to the camera and occupies more pixels.

Objects mapped to a physical meter scale, demonstrating size constancy across different depths.

Limitations

The current model relies on a flat ground plane assumption, which may fail in complex environments like staircases or uneven terrain. Additionally, some categories (e.g., boats) exhibit high size variance that remains difficult to resolve without further context.

Conclusion

By treating "filling in the blanks" as a recognition task and using scene context as a calibration tool, this work provides a robust path toward veridical size estimation. Future research will likely explore more complex scene layouts and full 3D mesh reconstruction to further enhance spatial understanding.

Summary updated

Abstract

We consider the problem of enriching current object detection systems with veridical object sizes and relative depth estimates from a single image. There are several technical challenges to this, such as occlusions, lack of calibration data and the scale ambiguity between object size and distance. These have not been addressed in full generality in previous work. Here we propose to tackle these issues by building upon advances in object recognition and using recently created large-scale datasets. We first introduce the task of amodal bounding box completion, which aims to infer the the full extent of the object instances in the image. We then propose a probabilistic framework for learning category-specific object size distributions from available annotations and leverage these in conjunction with amodal completion to infer veridical sizes in novel images. Finally, we introduce a focal length prediction approach that exploits scene recognition to overcome inherent scaling ambiguities and we demonstrate qualitative results on challenging real-world scenes.
BibTeX
@inproceedings{kar2015amodal,
  author = {Abhishek Kar and Shubham Tulsiani and João Carreira and Jitendra Malik},
  title = {{Amodal Completion and Size Constancy in Natural Scenes}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  doi = {10.1109/iccv.2015.23},
}
Figure