Rectification Reimagined: A Unified Mamba Model for Image Correction and Rectangling with Prompts

Linwei Qiu, Gongzhe Li

Abstract

UniRect is a unified framework that tackles four distinct image rectification tasks—portrait correction, wide-angle rectangling, stitched image rectangling, and rotation correction—using a single architecture. By combining a progressive warping model with efficient Mamba-based restoration and a Sparse Mixture-of-Experts (SMoE) strategy, it achieves state-of-the-art results while remaining efficient enough for potential deployment on edge devices.

Introduction

Modern smartphone photography is a feat of software engineering, but it still struggles with geometric "imperfections." Wide-angle lenses distort faces at the edges of a frame; panoramas often result in "wavy" irregular boundaries; and handheld shots are frequently tilted. Currently, these issues are treated as separate problems, each requiring a specialized, task-specific deep learning model. This approach is inefficient for mobile devices with limited memory and ignores the underlying mathematical reality: all these problems are essentially different flavors of image rectification. The authors identify a major gap: the lack of a unified, task-agnostic model that can handle diverse distortions without "task competition"—a phenomenon where a model's performance on one task degrades its performance on another.

Methods

The authors introduce UniRect, which consists of three core innovations:

  1. General Distortion Model: They mathematically unify four different distortion types into a single equation, treating them as inverse problems of a general lens simulation.
  2. Dual-Component Pipeline:
    • Deformation Module (DM): Uses a Residual Progressive Thin-Plate Spline (RP-TPS) model. Instead of warping the image in one giant leap, it predicts control point locations progressively to minimize interpolation errors and handle complex geometric shifts.
  3. Restoration Module (RM): Warping an image often introduces artifacts or blur. To fix this, they use Residual Mamba Blocks (RMBs). Based on State Space Models (SSMs), Mamba provides the global context-awareness of Transformers but with linear computational complexity, making it ideal for high-resolution image restoration.
The UniRect architecture features a Deformation Module for progressive warping and a Restoration Module powered by Mamba blocks for artifact removal.
  1. Visual Prompts & SMoE: To handle multiple tasks at once, the model uses visual prompts (like face masks or boundary maps) to tell the network where to focus. To prevent different tasks from interfering with each other during training, they implement a Sparse Mixture-of-Experts (SMoEs) structure. A gating network dynamically assigns weights to different "expert" layers based on the input task.

Key Insights and Ideas

  • Geometric Unity: The most significant insight is the mathematical proof that wide-angle, stitched, and rotation distortions can be subsumed under a single general model. This justifies the move away from fragmented, task-specific architectures.
  • Mamba for Geometry: While Transformers are the current standard for global dependencies, their quadratic complexity is a bottleneck for high-res images. The authors show that Mamba’s scan characteristic is uniquely suited for capturing geometric information like image borders and long-range distortions efficiently.
  • Prompting as Decoupling: Using t-SNE visualizations, the authors demonstrate that visual prompts effectively "decouple" the data distributions of different tasks. This allows a single model to learn distinct behaviors for portraits versus panoramas without getting confused.
  • The Power of "Progressive" Warping: They found that simply applying a Thin-Plate Spline (TPS) once wasn't enough. By predicting residuals progressively, the model avoids the "chaotic pixel locations" that often plague single-step geometric corrections.

Key Results

The model was evaluated on benchmarks for portrait correction, wide-angle rectangling, stitched image rectangling, and rotation correction.

  • State-of-the-Art (SOTA): UniRect outperforms existing specialized models across almost all metrics (PSNR, SSIM, FID, and LPIPS). For example, in stitched image rectangling, it achieved a PSNR of 25.10 dB, a massive jump over the previous best of 21.28 dB.
  • Multi-Task Efficiency: The "four-in-one" model (UniRect-SMoE) performs nearly as well as four separate models trained individually, proving that the SMoE strategy successfully mitigates task competition.
  • Speed and Scale: The model runs at 35.8 FPS on an NVIDIA V100, making it significantly faster than diffusion-based alternatives while maintaining a manageable parameter count (357.9M).
Qualitative comparison showing UniRect's superior performance in correcting portraits, rectangling wide-angle and stitched images, and correcting rotation.

Limitations and Future Work

While more efficient than Transformers or Diffusion models, 357M parameters is still heavy for low-end mobile hardware. The authors suggest that making a "UniRect-Lite" is a logical next step. Additionally, real-world images often suffer from multiple distortions simultaneously (e.g., a tilted wide-angle shot). Creating datasets and training regimes for these "compound" problems is a goal for future research. Finally, like most supervised methods, the model's performance is tied to the quality of the ground truth in the training sets.

Summary updated

Abstract

Image correction and rectangling are valuable tasks in practical photography systems such as smartphones. Recent remarkable advancements in deep learning have undeniably brought about substantial performance improvements in these fields. Nevertheless, existing methods mainly rely on task-specific architectures. This significantly restricts their generalization ability and effective application across a wide range of different tasks. In this paper, we introduce the Unified Rectification Framework (UniRect), a comprehensive approach that addresses these practical tasks from a consistent distortion rectification perspective. Our approach incorporates various task-specific inverse problems into a general distortion model by simulating different types of lenses. To handle diverse distortions, UniRect adopts one task-agnostic rectification framework with a dual-component structure: a Deformation Module, which utilizes a novel Residual Progressive Thin-Plate Spline (RP-TPS) model to address complex geometric deformations, and a subsequent Restoration Module, which employs Residual Mamba Blocks (RMBs) to counteract the degradation caused by the deformation process and enhance the fidelity of the output image. Moreover, a Sparse Mixture-of-Experts (SMoEs) structure is designed to circumvent heavy task competition in multi-task learning due to varying distortions. Extensive experiments demonstrate that our models have achieved state-of-the-art performance compared with other up-to-date methods.
BibTeX
@inproceedings{qiu2026rectification,
  author = {Linwei Qiu and Gongzhe Li},
  title = {{Rectification Reimagined: A Unified Mamba Model for Image Correction and Rectangling with Prompts}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i10.37811},
}
Figure