TL;DR
The paper introduces UniRect (Unified Rectification Framework), a single, task-agnostic deep learning model capable of handling multiple image correction and rectangling tasks simultaneously. By combining a progressive thin-plate spline deformation module with state-of-the-art Residual Mamba Blocks and a Sparse Mixture-of-Experts strategy, UniRect eliminates the need for separate, task-specific architectures while achieving state-of-the-art performance on edge-device budgets.
What Problem Does This Paper Address?
When you snap a photo on a smartphone, several geometric distortions can compromise the image: wide-angle lenses warp faces near the edges (portrait distortion), panoramic stitching leaves irregular, wavy borders (stitched distortion), and accidental tilts require rotation corrections that clip out valuable pixels.
Historically, computer vision researchers have treated these as completely separate problems, designing bespoke, task-specific networks for each. This approach is highly inefficient for edge devices like smartphones, which operate under strict memory and computational constraints. Furthermore, it ignores the underlying mathematical relationships between different types of lens and perspective distortions. The authors tackle this gap by proposing a unified framework that views all these problems through a single lens: geometric rectification.
Methods
UniRect breaks down the problem into a unified mathematical formulation and a dual-component network architecture:
- General Distortion Model: The authors mathematically unify classic camera models (like Kannala-Brandt for wide-angle lenses and Brown-Conrady for radial distortions) into a single equation capable of describing all four target distortions.
- Deformation Module (DM): Instead of simple mesh warping, UniRect uses a novel Residual Progressive Thin-Plate Spline (RP-TPS) model. It progressively predicts control point adjustments across multiple stages, preventing intermediate interpolation errors and handling highly non-linear warps.
- Restoration Module (RM): Geometric warping often introduces blur or sampling artifacts. To fix this, the authors employ Residual Mamba Blocks (RMBs). Leveraging the linear-time sequence modeling capabilities of the Mamba architecture, this module efficiently captures long-range dependencies and global context to restore high-fidelity textures.
- Visual Prompts: To guide the unified network, task-specific visual prompts (e.g., face masks for portraits, boundary masks for rectangling) are concatenated with the input image, helping the model focus on the right regions.
- Sparse Mixture-of-Experts (SMoEs): When training one model on multiple tasks simultaneously, "task competition" usually degrades performance due to conflicting gradient directions. The authors introduce an SMoE structure with a gating network to dynamically route features to specialized expert layers, mitigating multi-task interference.
Datasets
The model was trained and evaluated on four public benchmarks representing each task:
- Portrait Correction: Wide-angle portrait dataset from Tan et al. (2021).
- Rectified Wide-Angle Image Rectangling: Dataset from Liao et al. (2023).
- Stitched Image Rectangling: Panoramic stitching dataset from Nie et al. (2022).
- Rotation Correction: Tilt-correction dataset sourced from MS-COCO/ImageNet by Nie et al. (2023).
Key Insights and Ideas
- The Geometry-Mamba Match: Mamba's scan characteristic is uniquely suited for geometric tasks. It excels at tracing image borders and long-range structural lines far more efficiently than standard CNNs or heavy Transformers.
- Prompt-Driven Decoupling: Using visual prompts (like a binary mask of irregular borders) shifts the input data distribution, effectively "decoupling" different tasks so a single network structure can process them without confusion.
- The Multi-Task Penalty: Simply training a single network on all tasks simultaneously (Mixed Learning) causes massive performance drops. The authors map out the underlying relationships between distortions to show how tasks overlap and compete, justifying the need for a Sparse MoE architecture.
Key Results
- State-of-the-Art Performance: UniRect outperforms or matches specialized single-task networks across all four domains.
- Quantitative Boosts: On stitched image rectangling ($\mathcal{T}_3$), UniRect achieved a PSNR of 25.10 dB, a massive leap over the previous best of 21.28 dB. For rotation correction ($\mathcal{T}_4$), it reached 23.16 dB PSNR, outperforming CoupledTPS (22.29 dB).
- Efficiency: The Mamba-based architecture balances accuracy and efficiency well, achieving 35.8 FPS on an NVIDIA V100 GPU with significantly fewer parameters than Transformer-based alternatives.
Limitations and Future Work
- Computational Overhead: While more efficient than Transformers, the restoration module is still heavy (>300M parameters) for ultra-low-resource edge deployment. The authors note that compressing this module is a key next step.
- Single Distortion Assumption: Current datasets only feature one type of distortion at a time. Real-world photos often suffer from multiple simultaneous issues (e.g., a tilted wide-angle portrait). Verifying and adapting UniRect to handle compound distortions remains an open challenge.
Summary updated