Enhanced Self-Supervised Multi-Image Super-Resolution for Camera Array Images

Yating Chen, Feng Huang, Xianyu Wu, Jing Wu, Ying ShenarXiv 2026 · DOI · arXiv

TL;DR

This paper introduces a novel self-supervised learning framework, Multi-to-Single-Guided Multi-to-Multi (GM2M), for enhancing the resolution of images captured by camera arrays. By combining the strengths of two existing self-supervised approaches, GM2M achieves superior detail restoration and reduces artifacts compared to prior methods, offering a more robust and efficient solution for camera array super-resolution.

What Problem Does This Paper Address?

Conventional multi-image super-resolution (MISR) methods often rely on sequential frames from a single camera, making them susceptible to complex degradations like severe occlusion and challenging image restoration. While camera arrays offer advantages like distributed views and reduced occlusion, existing MISR algorithms struggle to fully exploit these properties. Supervised methods tend to overfit to specific degradation patterns, and current self-supervised learning (SSL) techniques often fail to recover fine-grained details or can produce undesirable artifacts like "zipper edges." This paper aims to bridge this gap by developing an effective SSL approach for camera array images that can recover high-fidelity details without requiring paired low-resolution (LR) and high-resolution (HR) ground truth data.

Methods

The core of this work is the proposed Multi-to-Single-Guided Multi-to-Multi (GM2M) SSL framework. This framework integrates two SSL strategies:

  • Multi-to-Single SSL: This approach uses a single LR image as the target for training, leading to smooth edges and good visual perception but often sacrificing fine details.
  • Multi-to-Multi SSL: This method uses all LR images as targets, which can enhance edge sharpness and detail recovery but is prone to producing "zipper edges," especially with significant offsets between images.

The GM2M framework combines these by using the output of a Multi-to-Single trained network as an auxiliary supervised target to guide a Multi-to-Multi trained network. This allows the GM2M framework to leverage the edge-smoothing properties of Multi-to-Single while benefiting from the detail enhancement of Multi-to-Multi, resulting in images that are both detailed and visually appealing.

The paper also introduces a novel dual self-attention Transformer network called CASR-DSAT designed specifically for camera array SR. This network incorporates:

  • Channel Self-Attention Transformer Backbone (CSATB): For feature extraction, capturing global information and long-range dependencies.
  • Spatial Self-Attention Transformer Backbone (SSATB): For feature reconstruction, enabling larger non-local receptive fields to recover high-frequency details and mitigate noise.

The GM2M framework uses two CASR-DSAT networks, one trained with Multi-to-Single loss and another guided by it, with parameters optimized alternately. During testing, only the second network is used.

Datasets

  • Synthetic Training Dataset: Generated from the Zurich raw to RGB dataset, with LR sequences created by applying random translations, rotations, downsampling, and noise to HR images.
  • Synthetic Test Datasets:
    • DIV2K (natural scene images)
    • PROBA-V (satellite images)
    • Medical Dataset (MRI images)
  • Real Camera Array Datasets: Captured using two constructed nine-aperture camera-array imaging systems (one with Basler lenses, one with Edmund lenses). These datasets include outdoor natural scenes and indoor scenes with resolution targets.

Key Insights and Ideas

  • Synergy of SSL Approaches: The paper demonstrates that combining Multi-to-Single and Multi-to-Multi SSL strategies can overcome the individual limitations of each, leading to a more robust and effective SR solution.
  • GM2M as a New Paradigm: The GM2M framework offers a novel way to integrate deep neural networks with classical physics-based variational methods, enabling iterative parameter updates that improve efficiency.
  • Dual Self-Attention for Camera Arrays: The CASR-DSAT network, with its combination of channel and spatial self-attention Transformers, is specifically designed to handle the unique challenges of camera array imaging, such as aliasing artifacts.
  • Importance of Spatial Self-Attention for Reconstruction: The SSATB component, utilizing spatial self-attention, is crucial for recovering fine details and eliminating artifacts that simpler methods might miss.
  • SSL's Advantage for Real-World Data: The effectiveness of SSL is highlighted by its ability to perform well on real-world data without requiring ground truth, which is often difficult to obtain.

Key Results

  • Synthetic Data Performance: CASR-DSAT (both M2S and GM2M variants) achieved PSNR gains of over 1.0 dB compared to suboptimal methods across various synthetic datasets (DIV2K, PROBA-V, Medical). CASR-DSATGM2M showed superior LPIPS, indicating better texture restoration.
  • Real Data Performance: On real camera array data, CASR-DSAT accurately restored fine textures and details, outperforming other DL methods. While ML (a physics-based method) achieved higher SR magnification, it suffered from severe noise and artifacts.
  • GM2M vs. M2M: The GM2M framework significantly improved PSNR over the Multi-to-Multi SSL by 3.94 dB (DIV2K), 3.71 dB (PROBA-V), and 5.34 dB (Medical), while also eliminating zipper artifacts.
  • Network Efficiency: The proposed CASR-DSAT network has a parameter size of 14.45M, which is significantly smaller than other advanced models like AFCNet (47M), while achieving improved performance.

Limitations and Future Work

  • Motion Compensation Limitations: The paper acknowledges that the FNet used for motion compensation, being a fixed-kernel, fully convolutional optical flow estimation network, might struggle with complex occlusions in real-world scenes.
  • Future Work: The authors plan to explore spatially adaptive deformable convolutions to improve motion compensation flexibility and handle complex occlusions more effectively in future work.

Summary updated

BibTeX
@misc{chen2026enhanced,
  author = {Yating Chen and Feng Huang and Xianyu Wu and Jing Wu and Ying Shen},
  title = {{Enhanced Self-Supervised Multi-Image Super-Resolution for Camera Array Images}},
  year = {2026},
  eprint = {2604.06816},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.06816},
}
Figure