Towards Complete Scene and Regular Shape for Distortion Rectification by Curve-Aware Extrapolation

Kang Liao, Chunyu Lin, Yunchao Wei, Feng Li, Shangrong Yang, Yao Zhao

Introduction

Wide-angle lenses are essential for capturing expansive environments, yet they introduce significant radial distortion, commonly known as the "fisheye" effect. Traditional rectification techniques face a fundamental trade-off: "Rectangle Construction" crops the image, discarding peripheral data, while "Invagination Construction" preserves all pixels but results in irregular black borders.

This paper introduces Rectification OutPainting (ROP), a framework designed to achieve a "Complete Scene" and a "Regular Shape" simultaneously. By treating rectification as a specialized outpainting problem, ROP hallucinates missing pixels at the boundaries to produce a standard rectangular image with a field-of-view (FoV) that exceeds the original lens's physical capture.

Overview of the ROP process, showing the transition from a distorted wide-angle image to a complete, extrapolated rectangular result with increased FoV.

The ROP Framework

The ROP architecture is divided into two primary stages: the Distortion Rectification Module (DRM) and the Outpainting Module (OM).

Distortion Rectification Module (DRM)

The DRM utilizes a deep backbone (e.g., VGG or ResNet) to estimate distortion parameters from the input image. These parameters are used to warp the image into its "invaginated" form. A key innovation is the use of General Geometry Supervision, where the network learns to relate visual features directly to the curvature of the image boundaries.

Outpainting Module (OM)

The OM is a Generative Adversarial Network (GAN) optimized for filling the irregular voids created during rectification. It incorporates several novel mechanisms:

  • Dual Conditional Expansion: The generator is guided by two distinct critics. The Curvature Conditioned Critic ensures geometric continuity by matching the hallucinated area to the lens's geometric flow. The Semantic Conditioned Critic leverages pre-trained InceptionV3 features to ensure the new content is semantically plausible.
  • Curve-Aware Correlation Measurement: Unlike standard outpainting that assumes straight edges, ROP calculates pixel importance based on distance from curved boundaries, facilitating smoother transitions.
  • Dilated Residual Connections (DRC): These replace standard skip connections to capture long-range spatial context, preventing the blurring often seen in generative tasks.
Detailed architecture of the ROP framework, highlighting the Distortion Rectification Module and the dual-critic Outpainting Module.

Key Insights and Results

The core of the ROP approach is using boundary curvature as a bridge. The degree of "pinching" in the rectified image provides a mathematical signal for the network to determine the extent of distortion and the required hallucination area. This method effectively increases the FoV by approximately 22%.

Quantitative and Qualitative Performance

In benchmarks against state-of-the-art models like DeepFill v2 and HiFill, ROP consistently demonstrated superior performance. On synthetic datasets derived from MS-COCO, ROP achieved a PSNR of 23.40 on "Easy" cases, significantly outperforming competitors. Qualitatively, ROP maintains sharp textures and structural integrity (e.g., straight building lines) in the extrapolated regions, avoiding the artifacts and "seams" typical of other methods.

Downstream Utility

The practical benefits of ROP extend to computer vision tasks. The authors demonstrated that running Mask R-CNN on ROP-rectified images yields higher accuracy. By completing objects that were previously truncated or distorted at the edges, ROP provides a more reliable input for object detection and classification algorithms.

Conclusion and Future Work

ROP represents a significant step forward in wide-angle image processing by combining geometric rectification with generative outpainting. While the model excels at structural and textural hallucination, future work will address the challenge of generating highly complex, unique semantic objects at the frame edges. Additionally, optimizing the framework for real-time performance remains a priority for applications in autonomous systems and virtual reality.

Summary updated

Abstract

The wide-angle lens gains increasing attention since it can capture a wide field-of-view (FoV) scene. However, the obtained image is contaminated with radial distortion, making the scene not realistic. Previous distortion rectification methods rectify the image in a rectangle or invagination, failing to display the complete content and regular shape simultaneously. In this paper, we rethink the representation of rectification results and present a Rectification OutPainting (ROP) method, aiming to extrapolate the coherent semantics to the blank area and create a wider FoV beyond the original wide-angle lens. To address the specific challenges such as the variable painting region and curve boundary, a rectification module is designed to rectify the image with geometry supervision, and the extrapolated results are generated using a dual conditional expansion strategy. In terms of the spatially discounted correlation, a curve-aware correlation measurement is proposed to focus on the generated region to enforce the local consistency. To our knowledge, we are the first to tackle the challenging rectification via outpainting, and our curve-aware strategy can reach a rectification construction with complete content and regular shape. Extensive experiments well demonstrate the superiority of our ROP over other state-of-the-art solutions.
BibTeX
@inproceedings{liao2021towards,
  author = {Kang Liao and Chunyu Lin and Yunchao Wei and Feng Li and Shangrong Yang and Yao Zhao},
  title = {{Towards Complete Scene and Regular Shape for Distortion Rectification by Curve-Aware Extrapolation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01430},
}
Figure