Multi-Level Curriculum for Training A Distortion-Aware Barrel Distortion Rectification Model

Kang Liao, Chunyu Lin, Lixin Liao, Yao Zhao, Weiyao Lin

Introduction

Wide-angle and fish-eye lenses are essential for capturing expansive scenes in applications like autonomous driving and surveillance. However, they introduce significant "barrel distortion," where straight lines appear curved. While traditional feature-based methods struggle in environments lacking clear lines, recent deep learning approaches often fail to capture the complex, non-linear mapping required for high-quality rectification. This paper addresses these challenges by introducing a hierarchical framework guided by a multi-level curriculum.

Hierarchical Rectification Framework

The core of the proposed approach is a Multi-Level Curriculum that breaks down the rectification task into three progressive stages, analogous to a human artist's workflow: starting with a structural sketch, adding semantic coloring, and finally rendering fine textures.

The multi-level curriculum mimics the human painting process (a) and simplifies the learning task by focusing on visually observable distortion levels (b).

The framework consists of specialized modules:

  1. Structure Recovery: Focuses on fixing the geometric skeleton and contours.
  2. Semantics Embedding: Corrects the shapes and boundaries of high-level objects.
  3. Texture Rendering: Produces the final high-resolution RGB image.

To handle the spatial misalignment between distorted inputs and rectified outputs, the authors introduce the Location-Aligned Connection (LAC). Unlike standard skip connections in U-Nets, LAC "revises" encoder feature maps to align with the decoder's geometry, ensuring more accurate feature fusion.

Distortion-Aware Pre-training

Standard pre-training on datasets like ImageNet is often suboptimal for geometric tasks because it focuses on classification rather than spatial transformations. The authors propose a Distortion-Aware Pre-training strategy. Instead of predicting obscure parametric labels, the model is trained to estimate a visually observable "distortion level" ($D_{max}$).

Architecture of the distortion-aware pre-training phase, utilizing structure, semantics, and image encoders to estimate distortion levels.

This pre-training phase teaches the encoders to perceive the severity of the warp across different levels of abstraction (structure, semantics, and texture) before the rectification training begins.

Experimental Results

The model was trained on a synthetic dataset derived from ADE20K and evaluated across multiple domains, including COCO, Place365, and real-world footage from GoPro and other wide-angle cameras.

The proposed method achieved a PSNR of 26.71 and an SSIM of 0.88, significantly outperforming the previous state-of-the-art (Liao et al. 2020), which achieved 23.02 and 0.71, respectively. Qualitatively, the model excels at restoring straight lines and maintaining natural global scene distributions, even in complex real-world scenarios without specific camera calibration.

Conclusion

By adopting a "geometry-first" approach through a multi-level curriculum and distortion-aware pre-training, this work sets a new benchmark for barrel distortion rectification. While the model's reliance on semantic segmentation presents challenges for cross-domain generalization, it provides a robust foundation for future research in spatially-aware image restoration.

Summary updated

Abstract

Barrel distortion rectification aims at removing the radial distortion in a distorted image captured by a wide-angle lens. Previous deep learning methods mainly solve this problem by learning the implicit distortion parameters or the nonlinear rectified mapping function in a direct manner. However, this type of manner results in an indistinct learning process of rectification and thus limits the deep perception of distortion. In this paper, inspired by the curriculum learning, we analyze the barrel distortion rectification task in a progressive and meaningful manner. By considering the relationship among different construction levels in an image, we design a multi-level curriculum that disassembles the rectification task into three levels, structure recovery, semantics embedding, and texture rendering. With the guidance of the curriculum that corresponds to the construction of images, the proposed hierarchical architecture enables a progressive rectification and achieves more accurate results. Moreover, we present a novel distortion-aware pre-training strategy to facilitate the initial learning of neural networks, promoting the model to converge faster and better. Experimental results on the synthesized and real-world distorted image datasets show that the proposed approach significantly outperforms other learning methods, both qualitatively and quantitatively.
BibTeX
@inproceedings{liao2021multi,
  author = {Kang Liao and Chunyu Lin and Lixin Liao and Yao Zhao and Weiyao Lin},
  title = {{Multi-Level Curriculum for Training A Distortion-Aware Barrel Distortion Rectification Model}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00435},
}
Figure