Abstract: Working with images, one often faces problems with incomplete or unclear information. Image inpainting can be used to restore missing image regions but focuses, however, on low-level image features such as pixel intensity, pixel gradient orientation, and color. This paper aims to recover semantic image features (objects and positions) in images. Based on published gated PixelCNNs, we demonstrate a new approach referred to as quadro-directional PixelCNN to recover missing objects and return probable positions for objects based on the context. We call this approach context-based image segment labeling (CBISL). The results suggest that our four-directional model outperforms one-directional models (gated PixelCNN) and returns a human-comparable performance.
Introduction
In scenarios such as autonomous driving, visual information is frequently obscured by obstacles. While traditional image inpainting techniques can reconstruct missing pixels based on local textures and colors, they often lack a high-level understanding of the scene's logic. For example, a model might fill a gap in a sidewalk with gray pixels without "understanding" that a sidewalk is a likely location for a pedestrian. The gap in current research lies in semantic context completion—the ability to recover high-level features like object positions based on the surrounding environment.
Methodology
The researchers developed Context-based Image Segment Labeling (CBISL) using two primary architectural innovations:
- Block Label Matrices (BLM): Instead of processing raw, high-resolution pixels, the system divides a segmented image into an $m \times n$ grid of blocks. Each block is assigned a single label based on the dominant object class within it. This approach condenses the image into a manageable matrix of semantic information, allowing the model to focus on object relationships rather than low-level noise.
- Quadro-directional PixelCNN: Standard PixelCNNs are autoregressive, generating data in a "raster scan" order (top-left to bottom-right). This creates a directional bias or "blind spot." To overcome this, the authors trained four separate gated PixelCNNs on the same data, rotated by 0, 90, 180, and 270 degrees. By ensembling these four models, the system can analyze an unknown region from all four directions simultaneously.
Datasets
The model was evaluated on two major computer vision datasets:
- Cityscapes: Focused on structured urban street scenes, using 3,000 images for training.
- COCO-Stuff: A diverse dataset containing over 118,000 training images featuring a wide variety of natural objects and backgrounds.
Key Results and Insights
The quadro-directional approach consistently outperformed standard one-directional models. On the Cityscapes dataset with a 5x5 unknown area, the 4-directional model achieved a Top-1 accuracy of 88.60%, compared to 82.32% for the standard gated PixelCNN. Even when the occluded area was increased to cover 39% of the image, the model maintained a Top-5 accuracy of over 90%.
A significant feature of CBISL is its generation of probability maps. Rather than providing a single guess, the model produces a heatmap of likely object positions. For instance, it can indicate exactly where a pedestrian would most likely be located based on the position of the sidewalk and road, even if the pedestrian is not visible in the original data. In comparative tests, the model's predictions were found to be nearly identical to human intuition, slightly outperforming average human users on the COCO-Stuff dataset (90% vs 89% accuracy).
Limitations and Future Work
While highly effective in structured environments like Cityscapes, the model faces challenges with the high variability of natural scenes in COCO-Stuff. Future research will focus on scaling the training data to capture more complex semantic coherences and expanding the grid to allow for image extrapolation. Additionally, integrating temporal consistency for video material could significantly enhance predictive safety features in autonomous systems.
Summary updated